Nov 02 14:57:02.481858 ip-10-0-142-141 systemd[1]: Starting Kubernetes Kubelet... Nov 02 14:57:02.930158 ip-10-0-142-141 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:57:02.930158 ip-10-0-142-141 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 14:57:02.930158 ip-10-0-142-141 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:57:02.930158 ip-10-0-142-141 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:57:02.934658 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934551 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:02.934658 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934653 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:02.934658 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934659 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:02.934658 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934663 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934668 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934673 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934678 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934682 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934685 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934689 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934693 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934699 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934706 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934711 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934715 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934719 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934723 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934727 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934731 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934735 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934739 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934742 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:02.934907 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934754 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934758 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934762 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934766 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934769 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934773 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934777 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934782 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934786 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934790 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934795 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934800 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934805 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934809 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934813 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934818 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934823 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934827 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934832 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:02.935698 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934836 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934843 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934849 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934855 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934860 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934865 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934869 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934874 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934878 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934883 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934887 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934892 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934896 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934901 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934905 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934909 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934913 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934917 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934921 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:02.936460 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934925 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934929 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934934 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934938 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934943 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934949 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934953 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934957 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934962 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934967 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934971 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934976 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934980 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934984 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934988 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934993 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.934997 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935001 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935005 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:02.937284 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935011 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935015 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935019 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935023 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935028 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935032 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935037 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935041 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935045 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935050 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935054 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935058 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935062 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935066 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935070 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935074 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935079 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935084 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935088 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:02.938112 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935093 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935099 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935103 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935108 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935113 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935117 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935121 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.935125 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936378 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936388 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936393 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936398 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936402 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936406 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936412 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936416 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936420 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936424 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936429 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:02.938695 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936435 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936441 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936447 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936451 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936455 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936459 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936463 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936467 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936471 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936476 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936480 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936486 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936490 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936495 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936500 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936505 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936509 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936513 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936517 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936521 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:02.939150 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936525 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936530 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936534 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936538 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936542 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936547 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936551 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936555 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936560 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936564 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936569 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936573 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936578 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936582 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936586 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936590 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936615 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936619 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936623 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936627 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:02.939811 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936631 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936635 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936640 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936645 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936649 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936654 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936658 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936662 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936667 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936671 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936675 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936679 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936684 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936688 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936692 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936696 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936700 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936704 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936708 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:02.940827 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936713 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936717 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936722 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936726 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936732 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936737 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936742 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936746 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936751 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936757 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936762 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936771 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936775 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936780 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936784 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936788 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936793 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936797 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936802 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:02.941680 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936807 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936813 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936817 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936822 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936826 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936830 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936835 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936839 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936843 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936847 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936851 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936855 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936860 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936864 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936868 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936872 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.936877 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937714 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937723 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:02.942373 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937727 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937731 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937736 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937741 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937745 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937750 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937754 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937759 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937763 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937768 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937772 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937777 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937781 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937786 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937790 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937794 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937798 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937803 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937807 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:02.942852 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937811 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937815 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937820 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937824 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937829 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937833 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937837 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937842 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937846 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937850 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937855 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937859 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937864 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937868 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937873 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937877 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937881 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937885 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937889 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:02.943526 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937895 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937900 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937905 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937910 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937915 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937922 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937930 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937934 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937939 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937943 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937948 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937954 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937958 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937962 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937967 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937971 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937975 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937979 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937983 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:02.944022 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937987 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937992 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.937996 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938000 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938004 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938009 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938013 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938018 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938022 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938026 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938030 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938034 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938038 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938042 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938046 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938050 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938058 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938063 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938068 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938072 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:02.944467 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938076 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938081 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938085 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938089 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938094 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938098 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938102 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938106 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938110 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938115 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938119 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938123 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938127 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938131 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938135 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938139 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938144 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938148 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938154 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938158 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:02.945101 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938162 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938166 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938170 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938176 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938180 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938184 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.938188 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939420 2574 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939439 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939453 2574 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939459 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939466 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939472 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939479 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939486 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939491 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939496 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939502 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939507 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939513 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939518 2574 flags.go:64] FLAG: --cgroup-root="" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939523 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939528 2574 flags.go:64] FLAG: --client-ca-file="" Nov 02 14:57:02.945567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939532 2574 flags.go:64] FLAG: --cloud-provider="external" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939546 2574 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939554 2574 flags.go:64] FLAG: --cluster-domain="" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939559 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939564 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939569 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939574 2574 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939586 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939606 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939612 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939618 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939622 2574 flags.go:64] FLAG: --contention-profiling="false" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939628 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939633 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939638 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939642 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939649 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939656 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939662 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939667 2574 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939672 2574 flags.go:64] FLAG: --enable-server="true" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939676 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939684 2574 flags.go:64] FLAG: --event-burst="100" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939689 2574 flags.go:64] FLAG: --event-qps="50" Nov 02 14:57:02.946167 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939694 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939699 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939704 2574 flags.go:64] FLAG: --eviction-hard="" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939711 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939715 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939720 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939725 2574 flags.go:64] FLAG: --eviction-soft="" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939730 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939735 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939740 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939745 2574 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939749 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939754 2574 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939759 2574 flags.go:64] FLAG: --feature-gates="" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939766 2574 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939771 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939776 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939782 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939787 2574 flags.go:64] FLAG: --healthz-port="10248" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939792 2574 flags.go:64] FLAG: --help="false" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939797 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939802 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939808 2574 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 14:57:02.946802 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939812 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939818 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939826 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939833 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939837 2574 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939842 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939847 2574 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939852 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939857 2574 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939862 2574 flags.go:64] FLAG: --kube-reserved="" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939867 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939871 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939876 2574 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939881 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939887 2574 flags.go:64] FLAG: --lock-file="" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939891 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939896 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939901 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939910 2574 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939915 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939920 2574 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939925 2574 flags.go:64] FLAG: --logging-format="text" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939929 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939935 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 14:57:02.947347 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939940 2574 flags.go:64] FLAG: --manifest-url="" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939945 2574 flags.go:64] FLAG: --manifest-url-header="" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939951 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939957 2574 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939964 2574 flags.go:64] FLAG: --max-pods="110" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939969 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939974 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939979 2574 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939984 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939989 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.939995 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940002 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940012 2574 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940017 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940023 2574 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940028 2574 flags.go:64] FLAG: --pod-cidr="" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940032 2574 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940037 2574 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940042 2574 flags.go:64] FLAG: --pods-per-core="0" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940046 2574 flags.go:64] FLAG: --port="10250" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940051 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940056 2574 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0cf104e5ef24ae7f0" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940061 2574 flags.go:64] FLAG: --qos-reserved="" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940066 2574 flags.go:64] FLAG: --read-only-port="10255" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940071 2574 flags.go:64] FLAG: --register-node="true" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940076 2574 flags.go:64] FLAG: --register-with-taints="" Nov 02 14:57:02.947970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940082 2574 flags.go:64] FLAG: --registry-burst="10" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940086 2574 flags.go:64] FLAG: --registry-qps="5" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940091 2574 flags.go:64] FLAG: --reserved-cpus="" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940095 2574 flags.go:64] FLAG: --reserved-memory="" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940102 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940107 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940113 2574 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940117 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940121 2574 flags.go:64] FLAG: --runonce="false" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940126 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940132 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940136 2574 flags.go:64] FLAG: --seccomp-default="false" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940141 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940145 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940150 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940155 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940161 2574 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940168 2574 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940172 2574 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940177 2574 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940182 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940187 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940192 2574 flags.go:64] FLAG: --system-cgroups="" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940196 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940205 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 14:57:02.948848 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940210 2574 flags.go:64] FLAG: --tls-cert-file="" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940215 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940221 2574 flags.go:64] FLAG: --tls-min-version="" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940226 2574 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940230 2574 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940235 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940240 2574 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940244 2574 flags.go:64] FLAG: --v="2" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940251 2574 flags.go:64] FLAG: --version="false" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940257 2574 flags.go:64] FLAG: --vmodule="" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940263 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.940269 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940513 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940520 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940525 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940529 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940534 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940538 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940543 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940548 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940552 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940556 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:02.949554 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940561 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940567 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940573 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940579 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940584 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940588 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940612 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940617 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940621 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940625 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940629 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940634 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940638 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940642 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940647 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940651 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940655 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940660 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940664 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940668 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:02.950109 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940673 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940677 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940681 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940686 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940690 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940694 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940699 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940703 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940707 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940712 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940717 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940721 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940726 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940732 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940738 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940743 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940748 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940752 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940756 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940761 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:02.950619 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940772 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940777 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940781 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940785 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940789 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940793 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940798 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940803 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940807 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940811 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940816 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940820 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940824 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940828 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940832 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940836 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940842 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940848 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940853 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:02.951100 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940857 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940862 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.940866 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941301 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941308 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941312 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941316 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941321 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941325 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941329 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941334 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941339 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941343 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941348 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941352 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941356 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941360 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941365 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941369 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941373 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:02.951549 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941378 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941384 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941391 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941396 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941401 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941406 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941410 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941414 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941419 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941423 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941427 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941431 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941435 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941439 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941444 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941448 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:02.952038 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941453 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.941502 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941558 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941566 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941572 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941577 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941581 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941586 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941606 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941611 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941615 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941619 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941623 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941627 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941631 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941635 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:02.952402 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941640 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941644 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941647 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941652 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941656 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941660 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941664 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941669 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941673 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941677 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941681 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941685 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941689 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941693 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941697 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941701 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941705 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941711 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941715 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:02.952796 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941720 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941725 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941729 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941733 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941737 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941741 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941745 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941749 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941754 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941760 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941767 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941774 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941778 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941783 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941787 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941791 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941796 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941800 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941804 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:02.953238 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941808 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941812 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941816 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941821 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941825 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941829 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941834 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941838 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941843 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941847 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941851 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941855 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941860 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941865 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941870 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941875 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941880 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941885 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941889 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:02.953734 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941894 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941898 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941902 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941906 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941910 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941915 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941919 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941923 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941927 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941932 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941936 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941940 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941944 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941949 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941953 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941957 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941961 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941965 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941969 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941973 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:02.954177 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941977 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941985 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941989 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.941993 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.942000 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.942004 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.942009 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.942014 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.942018 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.942024 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.942028 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.942032 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.942036 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.942040 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.942045 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.950285 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.950300 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.951069 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.951076 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 14:57:02.954659 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951135 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951140 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951144 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951147 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951150 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951153 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951156 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951159 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951161 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951164 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951166 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951169 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951172 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951174 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951177 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951180 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951182 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951185 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951187 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:02.955117 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951190 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951193 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951195 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951197 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951200 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951203 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951205 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951208 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951212 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951221 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951226 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951229 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951232 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951235 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951237 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951240 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951242 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951245 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951247 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:02.955556 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951250 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951252 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951255 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951258 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951260 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951263 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951265 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951268 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951270 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951273 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951275 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951278 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951282 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951285 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951288 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951291 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951294 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951297 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951300 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:02.956042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951302 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951305 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951307 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951310 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951313 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951316 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951319 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951321 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951324 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951326 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951329 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951331 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951334 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951336 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951339 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951341 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951344 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951347 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951349 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:02.956485 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951352 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951354 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951357 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951359 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951362 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951364 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951367 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951370 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951372 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951375 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951378 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951380 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951383 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951385 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951388 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951390 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951393 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951396 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951398 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:02.956995 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951401 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951404 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951406 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951409 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951411 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951414 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951416 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951419 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951422 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951424 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951427 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.951454 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951603 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951612 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951615 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951618 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:02.957450 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951621 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951624 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951627 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951630 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951632 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951635 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951637 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951640 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951642 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951645 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951648 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951650 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951653 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951656 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951658 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951661 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951663 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951666 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951668 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:02.957970 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951671 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951674 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951676 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951679 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951681 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951684 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951686 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951689 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951691 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951694 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951697 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951700 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951702 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951704 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951707 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951711 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951713 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951716 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951718 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951720 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:02.958424 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951723 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951725 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951728 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951730 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951733 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951737 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951742 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951746 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951749 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951751 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951754 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951757 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951759 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951762 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951765 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951768 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951770 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951773 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951775 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:02.959013 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951778 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951780 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951783 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951785 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951788 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951790 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951793 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951795 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951798 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951800 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951804 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951808 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951811 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951813 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951816 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951818 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951821 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951823 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951826 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:02.959484 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951829 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951832 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951835 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951837 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951840 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951843 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951845 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951848 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951851 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951854 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951856 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951858 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951861 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951863 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951866 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951869 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951871 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951874 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951876 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951879 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:02.960206 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951881 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951883 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951886 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951889 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951891 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.951915 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951966 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951971 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951974 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951977 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951979 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951982 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951985 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951988 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951991 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951993 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:02.960893 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951996 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.951998 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952001 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952003 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952006 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952008 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952011 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952013 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952016 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952018 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952021 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952023 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952026 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952028 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952031 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952033 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952035 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952038 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952041 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:02.961300 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952044 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952047 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952049 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952052 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952054 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952057 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952060 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952062 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952065 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952068 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952070 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952073 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952075 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952078 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952081 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952083 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952085 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952088 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952090 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952093 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:02.961978 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952095 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952098 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952100 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952103 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952105 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952107 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952110 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952112 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952115 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952117 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952121 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952125 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952128 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952130 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952133 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952136 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952138 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952141 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952148 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:02.962521 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952151 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952153 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952156 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952159 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952161 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952164 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952166 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952169 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952171 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952173 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952176 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952178 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952181 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952184 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952186 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952190 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952193 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952196 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952199 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:02.963175 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952201 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952204 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952206 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952209 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952211 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952214 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952216 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952219 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952221 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952224 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952227 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952230 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952233 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952236 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952238 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952243 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952245 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952248 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:02.952250 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:02.963704 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.952299 2574 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 14:57:02.964157 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.956482 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 14:57:02.964157 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.957543 2574 server.go:1014] "Starting client certificate rotation" Nov 02 14:57:02.964157 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.957631 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:57:02.964157 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.957678 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:57:02.964157 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.959786 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:57:02.964157 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.962928 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:57:02.974386 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.974371 2574 log.go:25] "Validated CRI v1 runtime API" Nov 02 14:57:02.979878 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.979864 2574 log.go:25] "Validated CRI v1 image API" Nov 02 14:57:02.980704 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.980676 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 14:57:02.983480 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.983461 2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ec76a8f-cadb-4acf-9be3-29e302aa1b3e:/dev/nvme0n1p3 fcd1a2af-8fc8-4a67-bb90-768d514bcbd0:/dev/nvme0n1p4] Nov 02 14:57:02.983534 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.983480 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 14:57:02.989023 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.988911 2574 manager.go:217] Machine: {Timestamp:2026-11-02 14:57:02.98744775 +0000 UTC m=+0.374829160 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100108 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec237768f56c812721c45ac86f21b54f SystemUUID:ec237768-f56c-8127-21c4-5ac86f21b54f BootID:eb039694-a253-48d0-b78c-910df1e21446 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d2:ec:15:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d2:ec:15:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:9c:b4:8e:8e:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 14:57:02.989740 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.989730 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 14:57:02.989856 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.989843 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 14:57:02.993568 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.993547 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 14:57:02.993738 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.993569 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-141.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 14:57:02.993787 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.993742 2574 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 14:57:02.993787 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.993750 2574 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 14:57:02.993787 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.993783 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:57:02.993864 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.993801 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:57:02.995096 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.995085 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 14:57:02.999783 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.999771 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 14:57:02.999901 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:02.999891 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 14:57:03.000771 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.000754 2574 kubelet.go:493] "Attempting to sync node with API server" Nov 02 14:57:03.000817 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.000774 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 14:57:03.000817 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.000789 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 14:57:03.000817 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.000799 2574 kubelet.go:405] "Adding apiserver pod source" Nov 02 14:57:03.000817 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.000807 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 14:57:03.003130 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.003104 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:57:03.006101 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.006079 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 14:57:03.012936 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.012924 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:57:03.013805 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.013794 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 14:57:03.013860 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.013828 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 14:57:03.013973 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.013964 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 14:57:03.014004 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.013978 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 14:57:03.014004 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.013984 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 14:57:03.014004 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.013990 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 14:57:03.014004 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.013995 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 14:57:03.014004 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.014000 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 14:57:03.014714 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.014706 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 14:57:03.014714 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.014714 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 14:57:03.014775 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.014722 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 14:57:03.014775 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.014728 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 14:57:03.014775 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.014737 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 14:57:03.014775 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.014745 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 14:57:03.016301 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.016292 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 14:57:03.016301 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.016301 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 14:57:03.020087 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.020074 2574 server.go:1263] "Started kubelet" Nov 02 14:57:03.020185 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.020150 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 14:57:03.020250 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.020178 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 14:57:03.020250 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.020242 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 14:57:03.020846 ip-10-0-142-141 systemd[1]: Started Kubernetes Kubelet. Nov 02 14:57:03.022070 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.022051 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 14:57:03.025101 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.025086 2574 server.go:317] "Adding debug handlers to kubelet server" Nov 02 14:57:03.026627 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.026609 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 14:57:03.029888 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.029869 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 14:57:03.030566 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.030551 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 14:57:03.030766 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.030745 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:57:03.031248 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031222 2574 factory.go:55] Registering systemd factory Nov 02 14:57:03.031248 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031241 2574 factory.go:223] Registration of the systemd container factory successfully Nov 02 14:57:03.031408 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031312 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 14:57:03.031408 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031324 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 14:57:03.031408 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031311 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 14:57:03.031495 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031475 2574 factory.go:153] Registering CRI-O factory Nov 02 14:57:03.031495 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031484 2574 reconstruct.go:97] "Volume reconstruction finished" Nov 02 14:57:03.031495 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031491 2574 factory.go:223] Registration of the crio container factory successfully Nov 02 14:57:03.031573 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031535 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 14:57:03.031573 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031550 2574 factory.go:103] Registering Raw factory Nov 02 14:57:03.031573 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031492 2574 reconciler.go:29] "Reconciler: start to sync state" Nov 02 14:57:03.031573 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.031564 2574 manager.go:1199] Started watching for new ooms in manager Nov 02 14:57:03.032534 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.032521 2574 manager.go:319] Starting recovery of all containers Nov 02 14:57:03.042784 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.042763 2574 manager.go:324] Recovery completed Nov 02 14:57:03.047965 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.047951 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:03.049681 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.049654 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:03.049773 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.049697 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:03.049773 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.049712 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:03.050264 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.050246 2574 cpu_manager.go:226] "Starting" policy="none" Nov 02 14:57:03.050264 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.050257 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:03.050376 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.050260 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 14:57:03.050376 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.050348 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 14:57:03.050996 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.050979 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:03.051049 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.051012 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:03.051049 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.051027 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:03.052564 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.052549 2574 policy_none.go:50] "Start" Nov 02 14:57:03.052564 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.052565 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 14:57:03.052667 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.052578 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 14:57:03.054571 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.054561 2574 policy_none.go:44] "Start" Nov 02 14:57:03.087717 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.087673 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 14:57:03.092377 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.092364 2574 manager.go:341] "Starting Device Plugin manager" Nov 02 14:57:03.092453 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.092413 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 14:57:03.092453 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.092423 2574 server.go:87] "Starting device plugin registration server" Nov 02 14:57:03.092638 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.092627 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 14:57:03.092683 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.092639 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 14:57:03.092728 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.092712 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 14:57:03.092863 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.092794 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 14:57:03.092863 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.092808 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 14:57:03.093208 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.093188 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 14:57:03.093275 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.093234 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:57:03.160864 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.160171 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-141.us-west-2.compute.internal.18e44246b316a76c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-141.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-141.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 14:57:03.020050284 +0000 UTC m=+0.407431695,LastTimestamp:2026-11-02 14:57:03.020050284 +0000 UTC m=+0.407431695,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-141.us-west-2.compute.internal,}" Nov 02 14:57:03.162883 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.162862 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-141.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 14:57:03.164314 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.164291 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 14:57:03.165449 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.165428 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 14:57:03.165533 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.165453 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 14:57:03.165533 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.165475 2574 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 14:57:03.165533 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.165506 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 14:57:03.192910 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.192859 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:03.193647 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.193622 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:03.193718 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.193663 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:03.193718 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.193677 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:03.193718 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.193707 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.203527 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.203504 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-141.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 14:57:03.224504 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.224295 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-141.us-west-2.compute.internal.18e44246b4dac6df default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-141.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-141.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-141.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 14:57:03.049680607 +0000 UTC m=+0.437062023,LastTimestamp:2026-11-02 14:57:03.049680607 +0000 UTC m=+0.437062023,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-141.us-west-2.compute.internal,}" Nov 02 14:57:03.265622 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.265588 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal"] Nov 02 14:57:03.265699 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.265652 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:03.266615 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.266587 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:03.266692 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.266624 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:03.266692 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.266634 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:03.268895 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.268883 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:03.269041 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.269023 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.269099 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.269065 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:03.269821 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.269807 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:03.269900 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.269834 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:03.269900 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.269847 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:03.269900 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.269868 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:03.269900 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.269892 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:03.270125 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.269909 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:03.271962 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.271942 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.272041 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.271967 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:03.273350 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.273333 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:03.273440 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.273357 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:03.273440 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.273366 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:03.297997 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.297980 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" node="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.300632 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.300618 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" node="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.310502 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.310472 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 14:57:03.310583 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.310499 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 14:57:03.310660 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.310630 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 02 14:57:03.310844 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.310415 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-141.us-west-2.compute.internal.18e44246b4db212a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-141.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-141.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-141.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 14:57:03.049703722 +0000 UTC m=+0.437085134,LastTimestamp:2026-11-02 14:57:03.049703722 +0000 UTC m=+0.437085134,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-141.us-west-2.compute.internal,}" Nov 02 14:57:03.310948 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.310931 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-141.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 14:57:03.320410 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.320392 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 14:57:03.320479 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.320449 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ftxzz" Nov 02 14:57:03.333800 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.333776 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.333883 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.333801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d1881377e4cd46a1e5cc5db6f951a84-config\") pod \"kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"5d1881377e4cd46a1e5cc5db6f951a84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.333883 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.333819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.387187 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.387160 2574 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-142-141.us-west-2.compute.internal\" not found" node="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.434901 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.434872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.434901 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.434903 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.435096 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.434920 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d1881377e4cd46a1e5cc5db6f951a84-config\") pod \"kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"5d1881377e4cd46a1e5cc5db6f951a84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.435096 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.434951 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d1881377e4cd46a1e5cc5db6f951a84-config\") pod \"kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"5d1881377e4cd46a1e5cc5db6f951a84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.435096 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.434972 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.435096 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.434976 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.521101 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.521037 2574 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-141.us-west-2.compute.internal\" not found" node="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.587676 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.587638 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:03.588611 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.588576 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:03.588735 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.588621 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:03.588735 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.588631 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:03.588735 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.588658 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.600776 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.600753 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.604161 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.604142 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.644631 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:03.644612 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:03.644631 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.644635 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-141.us-west-2.compute.internal\": node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:57:03.760421 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:03.760388 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:57:05.283776 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:05.283746 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:57:05.380145 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:05.380107 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 14:57:05.380662 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:05.380294 2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae753dff0150c41c2b123ad91ad36b56-7d3113d1bc7bb9ad.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-142-141.us-west-2.compute.internal.18e44246b4dac6df\": read tcp 10.0.142.141:54888->13.58.87.20:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-142-141.us-west-2.compute.internal.18e44246b4dac6df default 5257 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-141.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-141.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-141.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 14:57:03 +0000 UTC,LastTimestamp:2026-11-02 14:57:03.266612243 +0000 UTC m=+0.653993653,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-141.us-west-2.compute.internal,}" Nov 02 14:57:05.384396 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:05.384379 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:57:05.486190 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:05.486109 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:57:05.586642 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:05.586615 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:57:05.687177 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:05.687138 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:57:05.716203 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:05.716178 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:05.717203 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:05.717183 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:05.717327 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:05.717217 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:05.717327 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:05.717230 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:05.747649 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:05.747570 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:52:04 +0000 UTC" deadline="2028-05-06 21:34:33.177245287 +0000 UTC" Nov 02 14:57:05.747649 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:05.747608 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13230h37m27.429641824s" Nov 02 14:57:05.751336 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:05.751318 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:05.787972 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:05.787935 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:57:05.888532 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:05.888498 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:57:05.961990 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:05.961963 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:05.962702 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:05.962686 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:06.053831 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.053753 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:06.114625 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.114598 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:57:06.115322 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.115308 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:57:06.178853 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.178826 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:57:06.222083 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:06.222053 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1881377e4cd46a1e5cc5db6f951a84.slice/crio-a75e8b0581b792e612a420f7c4bfdcb2db686b6af2d26b1ed651279f6dd9461a WatchSource:0}: Error finding container a75e8b0581b792e612a420f7c4bfdcb2db686b6af2d26b1ed651279f6dd9461a: Status 404 returned error can't find the container with id a75e8b0581b792e612a420f7c4bfdcb2db686b6af2d26b1ed651279f6dd9461a Nov 02 14:57:06.227071 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.227056 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:57:06.272557 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:06.272533 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20771b11b53d11f8bd2b40993a0c6e2b.slice/crio-4611b7d9a9374bac69debb4ab4817106ff8e10f9c8514bad2b93228be942d3e6 WatchSource:0}: Error finding container 4611b7d9a9374bac69debb4ab4817106ff8e10f9c8514bad2b93228be942d3e6: Status 404 returned error can't find the container with id 4611b7d9a9374bac69debb4ab4817106ff8e10f9c8514bad2b93228be942d3e6 Nov 02 14:57:06.276335 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.276318 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:06.425328 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.425251 2574 apiserver.go:52] "Watching apiserver" Nov 02 14:57:06.453492 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.453466 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 14:57:06.528294 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.528270 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 14:57:06.528574 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.528557 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-qpkcp","openshift-cluster-node-tuning-operator/tuned-nrz94","openshift-image-registry/node-ca-zdkx8","openshift-network-diagnostics/network-check-target-vxjb7","kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7","openshift-dns/node-resolver-kjh9p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-lqtnw","openshift-network-operator/iptables-alerter-xsbm4","kube-system/konnectivity-agent-xp4n5","openshift-multus/multus-grr2c","openshift-multus/network-metrics-daemon-f9zqb","kube-system/global-pull-secret-syncer-vjfjv"] Nov 02 14:57:06.531285 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.531266 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:57:06.533339 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.533317 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:06.535356 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.535338 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:57:06.537464 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.537384 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:06.537464 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:06.537445 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:06.539479 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.539463 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:06.539556 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:06.539524 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:06.541581 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.541562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.543652 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.543638 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.545867 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.545847 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:06.545938 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:06.545921 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:06.548909 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.548889 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.551244 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.551226 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:57:06.553409 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.553390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.555720 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.555696 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.566371 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-systemd-units\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.566444 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-netns\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.566444 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.566509 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566452 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-netd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.566509 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566476 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-cnibin\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.566570 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566507 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-modprobe-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.566570 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566550 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-k8s-cni-cncf-io\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.566570 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566567 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-conf-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.566686 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566581 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efc5a077-d018-4792-8551-b9c3594967dd-agent-certs\") pod \"konnectivity-agent-xp4n5\" (UID: \"efc5a077-d018-4792-8551-b9c3594967dd\") " pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:06.566686 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efc5a077-d018-4792-8551-b9c3594967dd-konnectivity-ca\") pod \"konnectivity-agent-xp4n5\" (UID: \"efc5a077-d018-4792-8551-b9c3594967dd\") " pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:06.566686 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-device-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.566808 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnvct\" (UniqueName: \"kubernetes.io/projected/b3d8b64c-559c-4806-8599-25b62c630fb4-kube-api-access-lnvct\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:57:06.566808 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29cc4\" (UniqueName: \"kubernetes.io/projected/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-kube-api-access-29cc4\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:06.566808 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-tmp\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.566808 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-bin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.566964 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-registration-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.566964 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3d8b64c-559c-4806-8599-25b62c630fb4-hosts-file\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:57:06.566964 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566887 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j698s\" (UniqueName: \"kubernetes.io/projected/2e7e4e7b-c190-420d-b438-b2c9e1074795-kube-api-access-j698s\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:57:06.566964 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-var-lib-kubelet\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.566964 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566944 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-slash\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.567197 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566969 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d8b64c-559c-4806-8599-25b62c630fb4-tmp-dir\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:57:06.567197 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.566985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-conf\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.567197 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567019 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.567197 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-multus\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.567197 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-daemon-config\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.567197 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567118 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.567197 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.567524 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.567524 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567278 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-kubelet-config\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:06.567524 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567302 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-config\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.567524 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-env-overrides\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.567524 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567374 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-script-lib\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.567524 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cnibin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.567524 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-kubernetes\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.567524 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-kubelet\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.567524 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p65ch\" (UniqueName: \"kubernetes.io/projected/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-kube-api-access-p65ch\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.567524 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567509 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567532 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-lib-modules\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4p4n\" (UniqueName: \"kubernetes.io/projected/8880c66c-32b2-4a81-a174-34242093784d-kube-api-access-r4p4n\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567605 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovn-node-metrics-cert\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5hh7\" (UniqueName: \"kubernetes.io/projected/449bb762-da7a-4a88-9796-1a1ef23e7c8c-kube-api-access-x5hh7\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-socket-dir-parent\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-hostroot\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ffbd00b-29ae-4d63-97b8-edf990224dcd-iptables-alerter-script\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567757 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-etc-tuned\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-multus-certs\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-sys-fs\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567882 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbr8c\" (UniqueName: \"kubernetes.io/projected/0444057f-0080-4135-9d83-dca94858033e-kube-api-access-wbr8c\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.567930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.567970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysconfig\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-sys\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-ovn\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568086 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-log-socket\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-bin\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-systemd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-node-log\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-system-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-var-lib-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-netns\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568345 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-kubelet\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568359 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e7e4e7b-c190-420d-b438-b2c9e1074795-serviceca\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.568541 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568408 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-etc-kubernetes\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568440 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hzs7\" (UniqueName: \"kubernetes.io/projected/4ffbd00b-29ae-4d63-97b8-edf990224dcd-kube-api-access-7hzs7\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e7e4e7b-c190-420d-b438-b2c9e1074795-host\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-host\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568520 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-etc-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cni-binary-copy\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568554 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-dbus\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-socket-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568602 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr7rp\" (UniqueName: \"kubernetes.io/projected/b5cc5088-ab54-47e3-bcca-7cb792f88880-kube-api-access-lr7rp\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568639 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ffbd00b-29ae-4d63-97b8-edf990224dcd-host-slash\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-system-cni-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-systemd\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.569082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.568698 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-run\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.591930 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.591892 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" event={"ID":"20771b11b53d11f8bd2b40993a0c6e2b","Type":"ContainerStarted","Data":"4611b7d9a9374bac69debb4ab4817106ff8e10f9c8514bad2b93228be942d3e6"} Nov 02 14:57:06.592861 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.592836 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" event={"ID":"5d1881377e4cd46a1e5cc5db6f951a84","Type":"ContainerStarted","Data":"a75e8b0581b792e612a420f7c4bfdcb2db686b6af2d26b1ed651279f6dd9461a"} Nov 02 14:57:06.592955 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.592916 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s2tzl\"" Nov 02 14:57:06.594874 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.594858 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 14:57:06.595064 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.595051 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 14:57:06.595468 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.595453 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 14:57:06.595610 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.595597 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.595684 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.595633 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.595684 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.595674 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dlx4j\"" Nov 02 14:57:06.595769 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.595693 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.595838 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.595827 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.596025 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.595979 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 14:57:06.596025 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.596006 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b9vbw\"" Nov 02 14:57:06.596116 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.596025 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.596116 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.596033 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.596628 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.596617 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tsn9p\"" Nov 02 14:57:06.601537 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.601517 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.601620 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.601545 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.601620 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.601551 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 14:57:06.601620 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.601591 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q62fl\"" Nov 02 14:57:06.601729 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.601661 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 14:57:06.601775 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.601763 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 14:57:06.605142 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.605131 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.633577 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.633559 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 14:57:06.633650 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.633608 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cftkm" Nov 02 14:57:06.655080 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.655061 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 14:57:06.669124 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-kubernetes\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.669192 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-kubelet\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.669192 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65ch\" (UniqueName: \"kubernetes.io/projected/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-kube-api-access-p65ch\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.669192 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.669292 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-lib-modules\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.669292 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669194 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-kubelet\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.669292 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669250 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-kubernetes\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.669385 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4p4n\" (UniqueName: \"kubernetes.io/projected/8880c66c-32b2-4a81-a174-34242093784d-kube-api-access-r4p4n\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.669385 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669289 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.669385 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669314 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-lib-modules\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.669385 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669330 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovn-node-metrics-cert\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.669385 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669358 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hh7\" (UniqueName: \"kubernetes.io/projected/449bb762-da7a-4a88-9796-1a1ef23e7c8c-kube-api-access-x5hh7\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.669385 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-socket-dir-parent\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-hostroot\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ffbd00b-29ae-4d63-97b8-edf990224dcd-iptables-alerter-script\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-etc-tuned\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669438 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-hostroot\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-multus-certs\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669487 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-socket-dir-parent\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-sys-fs\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669554 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbr8c\" (UniqueName: \"kubernetes.io/projected/0444057f-0080-4135-9d83-dca94858033e-kube-api-access-wbr8c\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669503 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-multus-certs\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysconfig\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.669630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669626 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-sys-fs\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669640 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-sys\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-ovn\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-log-socket\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysconfig\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669707 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:06.669715 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669719 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-bin\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669749 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669757 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-ovn\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669769 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-log-socket\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669770 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-sys\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:06.669790 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs podName:13823b41-3da3-4a0b-bf1d-4c9d1b24608f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:07.169769078 +0000 UTC m=+3.134751586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs") pod "network-metrics-daemon-f9zqb" (UID: "13823b41-3da3-4a0b-bf1d-4c9d1b24608f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669805 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-bin\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669822 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669836 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.670287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669847 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-systemd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669863 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-systemd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-node-log\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-system-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669910 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-var-lib-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669923 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-node-log\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669943 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-var-lib-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669950 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-netns\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669962 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-system-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-kubelet\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.669985 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-netns\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e7e4e7b-c190-420d-b438-b2c9e1074795-serviceca\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670028 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-kubelet\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-etc-kubernetes\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670137 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.671179 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670152 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-etc-kubernetes\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670181 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hzs7\" (UniqueName: \"kubernetes.io/projected/4ffbd00b-29ae-4d63-97b8-edf990224dcd-kube-api-access-7hzs7\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:06.670197 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670203 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e7e4e7b-c190-420d-b438-b2c9e1074795-host\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-host\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:06.670242 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret podName:4bfe09e4-8fce-4149-a9fb-0a222124f121 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:07.170226699 +0000 UTC m=+3.135209202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret") pod "global-pull-secret-syncer-vjfjv" (UID: "4bfe09e4-8fce-4149-a9fb-0a222124f121") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670269 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-host\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-etc-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670291 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-etc-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670300 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cni-binary-copy\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670301 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-dbus\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670336 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e7e4e7b-c190-420d-b438-b2c9e1074795-host\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-socket-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670417 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e7e4e7b-c190-420d-b438-b2c9e1074795-serviceca\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670417 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-dbus\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr7rp\" (UniqueName: \"kubernetes.io/projected/b5cc5088-ab54-47e3-bcca-7cb792f88880-kube-api-access-lr7rp\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.672037 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ffbd00b-29ae-4d63-97b8-edf990224dcd-host-slash\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670465 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-socket-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-system-cni-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670502 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ffbd00b-29ae-4d63-97b8-edf990224dcd-host-slash\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670510 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-system-cni-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-systemd\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-run\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670567 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-systemd-units\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-netns\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670602 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670638 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-systemd-units\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-netd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670654 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-systemd\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-cnibin\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670647 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-netns\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670690 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670696 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-netd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.672672 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-modprobe-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670716 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-run\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-k8s-cni-cncf-io\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-conf-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670733 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-cnibin\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670773 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efc5a077-d018-4792-8551-b9c3594967dd-agent-certs\") pod \"konnectivity-agent-xp4n5\" (UID: \"efc5a077-d018-4792-8551-b9c3594967dd\") " pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670784 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-modprobe-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670790 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-k8s-cni-cncf-io\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670794 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efc5a077-d018-4792-8551-b9c3594967dd-konnectivity-ca\") pod \"konnectivity-agent-xp4n5\" (UID: \"efc5a077-d018-4792-8551-b9c3594967dd\") " pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670792 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-conf-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-device-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670837 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnvct\" (UniqueName: \"kubernetes.io/projected/b3d8b64c-559c-4806-8599-25b62c630fb4-kube-api-access-lnvct\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670855 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cni-binary-copy\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670865 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-device-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29cc4\" (UniqueName: \"kubernetes.io/projected/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-kube-api-access-29cc4\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-tmp\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-bin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-registration-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.673186 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.670987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3d8b64c-559c-4806-8599-25b62c630fb4-hosts-file\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j698s\" (UniqueName: \"kubernetes.io/projected/2e7e4e7b-c190-420d-b438-b2c9e1074795-kube-api-access-j698s\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671085 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-bin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-var-lib-kubelet\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671123 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3d8b64c-559c-4806-8599-25b62c630fb4-hosts-file\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-slash\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d8b64c-559c-4806-8599-25b62c630fb4-tmp-dir\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-conf\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671234 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-var-lib-kubelet\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671185 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-registration-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671208 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-slash\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671303 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671305 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efc5a077-d018-4792-8551-b9c3594967dd-konnectivity-ca\") pod \"konnectivity-agent-xp4n5\" (UID: \"efc5a077-d018-4792-8551-b9c3594967dd\") " pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671364 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-multus\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-multus\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671395 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-conf\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.673858 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671408 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-daemon-config\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671489 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671499 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d8b64c-559c-4806-8599-25b62c630fb4-tmp-dir\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-kubelet-config\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671523 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-config\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671969 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.671559 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-env-overrides\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.672083 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-script-lib\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.672102 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-kubelet-config\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.672107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cnibin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.672160 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cnibin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.672398 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.673306 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-etc-tuned\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.673330 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-tmp\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.674323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.673633 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efc5a077-d018-4792-8551-b9c3594967dd-agent-certs\") pod \"konnectivity-agent-xp4n5\" (UID: \"efc5a077-d018-4792-8551-b9c3594967dd\") " pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:06.686609 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.686502 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2tgnv\"" Nov 02 14:57:06.686609 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.686505 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hnsn9\"" Nov 02 14:57:06.686609 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.686542 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ct4hh\"" Nov 02 14:57:06.686609 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.686578 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4crbl\"" Nov 02 14:57:06.686609 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.686601 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 14:57:06.691323 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.691302 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovn-node-metrics-cert\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.692979 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.692956 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.692979 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.693004 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.693150 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.693027 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.693150 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.693101 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 14:57:06.693150 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.693134 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 14:57:06.693150 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.693144 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 14:57:06.693306 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.693153 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:06.693306 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.693213 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 14:57:06.693306 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.693286 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 14:57:06.693386 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.693342 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 14:57:06.700870 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.700849 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ffbd00b-29ae-4d63-97b8-edf990224dcd-iptables-alerter-script\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:57:06.703223 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.703201 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-env-overrides\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.703298 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.703222 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-config\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.703298 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.703207 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-daemon-config\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.703643 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.703625 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-script-lib\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.729091 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:06.729067 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:06.729091 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:06.729084 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:06.729091 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:06.729093 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6mjmh for pod openshift-network-diagnostics/network-check-target-vxjb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:06.729302 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:06.729160 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh podName:d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:07.229145039 +0000 UTC m=+3.194127545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6mjmh" (UniqueName: "kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh") pod "network-check-target-vxjb7" (UID: "d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:06.731333 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.731314 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hzs7\" (UniqueName: \"kubernetes.io/projected/4ffbd00b-29ae-4d63-97b8-edf990224dcd-kube-api-access-7hzs7\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:57:06.731712 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.731694 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbr8c\" (UniqueName: \"kubernetes.io/projected/0444057f-0080-4135-9d83-dca94858033e-kube-api-access-wbr8c\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.731919 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.731893 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29cc4\" (UniqueName: \"kubernetes.io/projected/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-kube-api-access-29cc4\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:06.731963 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.731925 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4p4n\" (UniqueName: \"kubernetes.io/projected/8880c66c-32b2-4a81-a174-34242093784d-kube-api-access-r4p4n\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.732015 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.731964 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnvct\" (UniqueName: \"kubernetes.io/projected/b3d8b64c-559c-4806-8599-25b62c630fb4-kube-api-access-lnvct\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:57:06.732497 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.732478 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hh7\" (UniqueName: \"kubernetes.io/projected/449bb762-da7a-4a88-9796-1a1ef23e7c8c-kube-api-access-x5hh7\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.735694 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.735667 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65ch\" (UniqueName: \"kubernetes.io/projected/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-kube-api-access-p65ch\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.736025 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.735986 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr7rp\" (UniqueName: \"kubernetes.io/projected/b5cc5088-ab54-47e3-bcca-7cb792f88880-kube-api-access-lr7rp\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.736130 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.736108 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j698s\" (UniqueName: \"kubernetes.io/projected/2e7e4e7b-c190-420d-b438-b2c9e1074795-kube-api-access-j698s\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:57:06.843304 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.843275 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:57:06.848526 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.848504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:06.853025 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:06.853001 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d8b64c_559c_4806_8599_25b62c630fb4.slice/crio-c33a739d8882496964cbd135a96dbe0c794add472b1fa44e7ae8d4809d0dd321 WatchSource:0}: Error finding container c33a739d8882496964cbd135a96dbe0c794add472b1fa44e7ae8d4809d0dd321: Status 404 returned error can't find the container with id c33a739d8882496964cbd135a96dbe0c794add472b1fa44e7ae8d4809d0dd321 Nov 02 14:57:06.856413 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:06.856393 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc5a077_d018_4792_8551_b9c3594967dd.slice/crio-d38fb6ced2844cadde02cd689ad85b4ce843ca0b911e766acfa0be64aed8c918 WatchSource:0}: Error finding container d38fb6ced2844cadde02cd689ad85b4ce843ca0b911e766acfa0be64aed8c918: Status 404 returned error can't find the container with id d38fb6ced2844cadde02cd689ad85b4ce843ca0b911e766acfa0be64aed8c918 Nov 02 14:57:06.858293 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.858279 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:57:06.865925 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.865905 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:57:06.866648 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:06.866628 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7e4e7b_c190_420d_b438_b2c9e1074795.slice/crio-057b1b17adaf5c318b46dee2951fccd0646743349eb73780ec96eed240d7eba9 WatchSource:0}: Error finding container 057b1b17adaf5c318b46dee2951fccd0646743349eb73780ec96eed240d7eba9: Status 404 returned error can't find the container with id 057b1b17adaf5c318b46dee2951fccd0646743349eb73780ec96eed240d7eba9 Nov 02 14:57:06.873707 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:06.873689 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cc5088_ab54_47e3_bcca_7cb792f88880.slice/crio-6ee9fd38117b061d44bdb2dfc5cc601cbc938be4d997d0f5c323407dd3d8dd2f WatchSource:0}: Error finding container 6ee9fd38117b061d44bdb2dfc5cc601cbc938be4d997d0f5c323407dd3d8dd2f: Status 404 returned error can't find the container with id 6ee9fd38117b061d44bdb2dfc5cc601cbc938be4d997d0f5c323407dd3d8dd2f Nov 02 14:57:06.878049 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.878033 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:57:06.880113 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.880099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:57:06.883396 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.883384 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:57:06.887311 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:06.887290 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8880c66c_32b2_4a81_a174_34242093784d.slice/crio-f0eba64bfbc755861c916495e51867223232bad108880992b7622181aeb7ac83 WatchSource:0}: Error finding container f0eba64bfbc755861c916495e51867223232bad108880992b7622181aeb7ac83: Status 404 returned error can't find the container with id f0eba64bfbc755861c916495e51867223232bad108880992b7622181aeb7ac83 Nov 02 14:57:06.890715 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:06.890697 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffbd00b_29ae_4d63_97b8_edf990224dcd.slice/crio-b8371c5f9d1ab400e9147eec249f811122ae10c36857b0842a7a09ed2870e9ce WatchSource:0}: Error finding container b8371c5f9d1ab400e9147eec249f811122ae10c36857b0842a7a09ed2870e9ce: Status 404 returned error can't find the container with id b8371c5f9d1ab400e9147eec249f811122ae10c36857b0842a7a09ed2870e9ce Nov 02 14:57:06.893565 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.893548 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-grr2c" Nov 02 14:57:06.893705 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:06.893688 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0444057f_0080_4135_9d83_dca94858033e.slice/crio-8aa5f84021e0925b39030f1e4c7ff4a7eb3179705920bc780302f2915aa6205d WatchSource:0}: Error finding container 8aa5f84021e0925b39030f1e4c7ff4a7eb3179705920bc780302f2915aa6205d: Status 404 returned error can't find the container with id 8aa5f84021e0925b39030f1e4c7ff4a7eb3179705920bc780302f2915aa6205d Nov 02 14:57:06.901652 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:06.901634 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2a2b02_cc3e_4a70_9d4e_a7c2bf896752.slice/crio-36a5aeef46f3c7ff8afbb40a561ff833653c954f6763644ecedb4d2cb9044704 WatchSource:0}: Error finding container 36a5aeef46f3c7ff8afbb40a561ff833653c954f6763644ecedb4d2cb9044704: Status 404 returned error can't find the container with id 36a5aeef46f3c7ff8afbb40a561ff833653c954f6763644ecedb4d2cb9044704 Nov 02 14:57:06.901729 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:06.901673 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:06.909799 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:06.909783 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449bb762_da7a_4a88_9796_1a1ef23e7c8c.slice/crio-25f28cedced0149eb0f8df7da6abed2fea8378201dc05ef9f35438b58942de77 WatchSource:0}: Error finding container 25f28cedced0149eb0f8df7da6abed2fea8378201dc05ef9f35438b58942de77: Status 404 returned error can't find the container with id 25f28cedced0149eb0f8df7da6abed2fea8378201dc05ef9f35438b58942de77 Nov 02 14:57:07.175444 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.175370 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:07.175444 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.175420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:07.175621 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:07.175512 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:07.175621 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:07.175543 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:07.175621 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:07.175574 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs podName:13823b41-3da3-4a0b-bf1d-4c9d1b24608f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:08.175557195 +0000 UTC m=+4.140539684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs") pod "network-metrics-daemon-f9zqb" (UID: "13823b41-3da3-4a0b-bf1d-4c9d1b24608f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:07.175621 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:07.175597 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret podName:4bfe09e4-8fce-4149-a9fb-0a222124f121 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:08.175579633 +0000 UTC m=+4.140562142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret") pod "global-pull-secret-syncer-vjfjv" (UID: "4bfe09e4-8fce-4149-a9fb-0a222124f121") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:07.276165 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.276136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:07.276318 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:07.276298 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:07.276318 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:07.276318 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:07.276383 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:07.276328 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6mjmh for pod openshift-network-diagnostics/network-check-target-vxjb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:07.276383 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:07.276372 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh podName:d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:08.276359285 +0000 UTC m=+4.241341778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mjmh" (UniqueName: "kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh") pod "network-check-target-vxjb7" (UID: "d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:07.597590 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.597499 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-grr2c" event={"ID":"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752","Type":"ContainerStarted","Data":"36a5aeef46f3c7ff8afbb40a561ff833653c954f6763644ecedb4d2cb9044704"} Nov 02 14:57:07.598654 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.598618 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" event={"ID":"b5cc5088-ab54-47e3-bcca-7cb792f88880","Type":"ContainerStarted","Data":"6ee9fd38117b061d44bdb2dfc5cc601cbc938be4d997d0f5c323407dd3d8dd2f"} Nov 02 14:57:07.599829 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.599798 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kjh9p" event={"ID":"b3d8b64c-559c-4806-8599-25b62c630fb4","Type":"ContainerStarted","Data":"c33a739d8882496964cbd135a96dbe0c794add472b1fa44e7ae8d4809d0dd321"} Nov 02 14:57:07.601210 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.601183 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" event={"ID":"449bb762-da7a-4a88-9796-1a1ef23e7c8c","Type":"ContainerStarted","Data":"25f28cedced0149eb0f8df7da6abed2fea8378201dc05ef9f35438b58942de77"} Nov 02 14:57:07.602403 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.602380 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqtnw" event={"ID":"0444057f-0080-4135-9d83-dca94858033e","Type":"ContainerStarted","Data":"8aa5f84021e0925b39030f1e4c7ff4a7eb3179705920bc780302f2915aa6205d"} Nov 02 14:57:07.603667 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.603641 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xsbm4" event={"ID":"4ffbd00b-29ae-4d63-97b8-edf990224dcd","Type":"ContainerStarted","Data":"b8371c5f9d1ab400e9147eec249f811122ae10c36857b0842a7a09ed2870e9ce"} Nov 02 14:57:07.605293 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.605271 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrz94" event={"ID":"8880c66c-32b2-4a81-a174-34242093784d","Type":"ContainerStarted","Data":"f0eba64bfbc755861c916495e51867223232bad108880992b7622181aeb7ac83"} Nov 02 14:57:07.606683 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.606660 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zdkx8" event={"ID":"2e7e4e7b-c190-420d-b438-b2c9e1074795","Type":"ContainerStarted","Data":"057b1b17adaf5c318b46dee2951fccd0646743349eb73780ec96eed240d7eba9"} Nov 02 14:57:07.608065 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.608048 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xp4n5" event={"ID":"efc5a077-d018-4792-8551-b9c3594967dd","Type":"ContainerStarted","Data":"d38fb6ced2844cadde02cd689ad85b4ce843ca0b911e766acfa0be64aed8c918"} Nov 02 14:57:07.636266 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.636173 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:52:06 +0000 UTC" deadline="2028-05-12 21:41:12.85369967 +0000 UTC" Nov 02 14:57:07.636266 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:07.636224 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13374h44m5.217479724s" Nov 02 14:57:08.183136 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:08.183057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:08.183136 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:08.183115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:08.183448 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:08.183266 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:08.183448 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:08.183323 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret podName:4bfe09e4-8fce-4149-a9fb-0a222124f121 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:10.183304286 +0000 UTC m=+6.148286782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret") pod "global-pull-secret-syncer-vjfjv" (UID: "4bfe09e4-8fce-4149-a9fb-0a222124f121") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:08.183561 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:08.183532 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:08.183612 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:08.183572 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs podName:13823b41-3da3-4a0b-bf1d-4c9d1b24608f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:10.183559798 +0000 UTC m=+6.148542287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs") pod "network-metrics-daemon-f9zqb" (UID: "13823b41-3da3-4a0b-bf1d-4c9d1b24608f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:08.283706 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:08.283640 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:08.284019 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:08.283988 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:08.284103 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:08.284025 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:08.284103 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:08.284039 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6mjmh for pod openshift-network-diagnostics/network-check-target-vxjb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:08.284103 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:08.284095 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh podName:d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:10.284078259 +0000 UTC m=+6.249060748 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mjmh" (UniqueName: "kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh") pod "network-check-target-vxjb7" (UID: "d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:08.594615 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:08.594096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:08.594615 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:08.594234 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:08.594615 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:08.594316 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:08.594615 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:08.594389 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:08.594615 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:08.594444 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:08.594615 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:08.594513 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:08.636675 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:08.636633 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:52:06 +0000 UTC" deadline="2028-04-15 09:43:42.403159757 +0000 UTC" Nov 02 14:57:08.636675 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:08.636674 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12714h46m33.766490861s" Nov 02 14:57:10.205786 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:10.205541 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:10.206279 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:10.205820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:10.206279 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:10.206032 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:10.206279 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:10.206096 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret podName:4bfe09e4-8fce-4149-a9fb-0a222124f121 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:14.206076001 +0000 UTC m=+10.171058513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret") pod "global-pull-secret-syncer-vjfjv" (UID: "4bfe09e4-8fce-4149-a9fb-0a222124f121") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:10.206279 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:10.206181 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:10.206279 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:10.206216 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs podName:13823b41-3da3-4a0b-bf1d-4c9d1b24608f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:14.206205321 +0000 UTC m=+10.171187825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs") pod "network-metrics-daemon-f9zqb" (UID: "13823b41-3da3-4a0b-bf1d-4c9d1b24608f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:10.306511 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:10.306471 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:10.306775 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:10.306686 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:10.306775 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:10.306711 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:10.306775 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:10.306725 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6mjmh for pod openshift-network-diagnostics/network-check-target-vxjb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:10.306962 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:10.306783 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh podName:d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:14.306766669 +0000 UTC m=+10.271749163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mjmh" (UniqueName: "kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh") pod "network-check-target-vxjb7" (UID: "d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:10.589281 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:10.589255 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:10.589415 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:10.589390 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:10.589710 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:10.589693 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:10.589815 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:10.589796 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:10.589877 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:10.589856 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:10.589940 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:10.589923 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:10.644490 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:10.644452 2574 generic.go:356] "Generic (PLEG): container finished" podID="20771b11b53d11f8bd2b40993a0c6e2b" containerID="0b16f51cda23d2f1d2dce38caab4b591e190b72533e300c173887f2a0ca7afeb" exitCode=0 Nov 02 14:57:10.644630 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:10.644550 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" event={"ID":"20771b11b53d11f8bd2b40993a0c6e2b","Type":"ContainerDied","Data":"0b16f51cda23d2f1d2dce38caab4b591e190b72533e300c173887f2a0ca7afeb"} Nov 02 14:57:10.656658 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:10.656618 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" event={"ID":"5d1881377e4cd46a1e5cc5db6f951a84","Type":"ContainerStarted","Data":"bb7872ee1c8810b6836115e003a91b0dd5aee5c94142dbca144fa1d57aab4105"} Nov 02 14:57:11.670574 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:11.670539 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" event={"ID":"20771b11b53d11f8bd2b40993a0c6e2b","Type":"ContainerStarted","Data":"3dc896d80538fd4cfadaf91494507219c00978b8faee1a7ae42bffe1e8fad367"} Nov 02 14:57:11.683325 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:11.683289 2574 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="40afaf4ef2bd07cf3eb5e4a8d4e052c098547430a16d75f8d2000b42c19276c1" exitCode=0 Nov 02 14:57:11.683451 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:11.683404 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqtnw" event={"ID":"0444057f-0080-4135-9d83-dca94858033e","Type":"ContainerDied","Data":"40afaf4ef2bd07cf3eb5e4a8d4e052c098547430a16d75f8d2000b42c19276c1"} Nov 02 14:57:11.803854 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:11.803742 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" podStartSLOduration=5.803723221 podStartE2EDuration="5.803723221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:10.876496341 +0000 UTC m=+6.841478853" watchObservedRunningTime="2026-11-02 14:57:11.803723221 +0000 UTC m=+7.768705733" Nov 02 14:57:11.928013 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:11.927940 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" podStartSLOduration=5.927911281 podStartE2EDuration="5.927911281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:11.804610008 +0000 UTC m=+7.769592525" watchObservedRunningTime="2026-11-02 14:57:11.927911281 +0000 UTC m=+7.892893793" Nov 02 14:57:12.591147 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:12.591110 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:12.591293 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:12.591262 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:12.592072 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:12.592051 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:12.592198 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:12.592170 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:12.592380 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:12.592350 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:12.592483 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:12.592461 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:12.688098 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:12.688058 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zdkx8" event={"ID":"2e7e4e7b-c190-420d-b438-b2c9e1074795","Type":"ContainerStarted","Data":"779cf2f2fe39d4e04ac9cdfb0d1986a0d04ceaf2ea5ecc2eef9ff53d6672ca9e"} Nov 02 14:57:12.692648 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:12.692623 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xp4n5" event={"ID":"efc5a077-d018-4792-8551-b9c3594967dd","Type":"ContainerStarted","Data":"6d220f61ebb855c2d4ff945b7d701a976c16cff6fc59e4251db963d367a472b8"} Nov 02 14:57:12.924435 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:12.924386 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xp4n5" podStartSLOduration=3.227129781 podStartE2EDuration="7.9243513s" totalImagesPullingTime="4.697221519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:12.924260137 +0000 UTC m=+8.889242648" watchObservedRunningTime="2026-11-02 14:57:12.9243513 +0000 UTC m=+8.889333811" Nov 02 14:57:12.925008 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:12.924962 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zdkx8" podStartSLOduration=2.286655109 podStartE2EDuration="7.924955322s" totalImagesPullingTime="5.638300213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:12.805362198 +0000 UTC m=+8.770344710" watchObservedRunningTime="2026-11-02 14:57:12.924955322 +0000 UTC m=+8.889937835" Nov 02 14:57:13.699437 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:13.698954 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" event={"ID":"b5cc5088-ab54-47e3-bcca-7cb792f88880","Type":"ContainerStarted","Data":"14b0967304572a54d345ea7db4c6e98a42215e53b9ac7e6c745bd2776c4f77a1"} Nov 02 14:57:14.237460 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:14.237407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:14.237635 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:14.237472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:14.237635 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:14.237620 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:14.237710 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:14.237682 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret podName:4bfe09e4-8fce-4149-a9fb-0a222124f121 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.23766426 +0000 UTC m=+18.202646771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret") pod "global-pull-secret-syncer-vjfjv" (UID: "4bfe09e4-8fce-4149-a9fb-0a222124f121") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:14.238097 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:14.238079 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:14.238197 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:14.238121 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs podName:13823b41-3da3-4a0b-bf1d-4c9d1b24608f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.238111843 +0000 UTC m=+18.203094344 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs") pod "network-metrics-daemon-f9zqb" (UID: "13823b41-3da3-4a0b-bf1d-4c9d1b24608f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:14.338805 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:14.338767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:14.338965 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:14.338911 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:14.338965 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:14.338927 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:14.338965 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:14.338939 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6mjmh for pod openshift-network-diagnostics/network-check-target-vxjb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:14.339142 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:14.338989 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh podName:d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.338974641 +0000 UTC m=+18.303957131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mjmh" (UniqueName: "kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh") pod "network-check-target-vxjb7" (UID: "d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:14.588901 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:14.588818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:14.589076 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:14.588943 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:14.589076 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:14.589031 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:14.589199 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:14.589107 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:14.589199 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:14.589154 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:14.589302 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:14.589200 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:15.406538 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:15.406509 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 14:57:15.521785 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:15.521073 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:57:15.406532807Z","UUID":"0d684aba-2f53-4891-8b4f-fe808861b657","Handler":null,"Name":"","Endpoint":""} Nov 02 14:57:15.532349 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:15.532329 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 14:57:15.532464 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:15.532371 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 14:57:15.708245 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:15.708208 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-grr2c" event={"ID":"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752","Type":"ContainerStarted","Data":"add4a7100580e932a15ebf42eb0e6378b31d130d21f04df66b0ba5aff39a652b"} Nov 02 14:57:15.711040 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:15.710967 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" event={"ID":"b5cc5088-ab54-47e3-bcca-7cb792f88880","Type":"ContainerStarted","Data":"45a42f40842ca24afb7f93596374a2d67218ac9c7f349314252dc62365bd6c52"} Nov 02 14:57:15.713074 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:15.713046 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kjh9p" event={"ID":"b3d8b64c-559c-4806-8599-25b62c630fb4","Type":"ContainerStarted","Data":"03c87d53056786c210f0011b0e98a4686f4a1bec0b2653a2ab971ea79a7eae31"} Nov 02 14:57:15.829967 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:15.829907 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-grr2c" podStartSLOduration=2.604702359 podStartE2EDuration="10.829888887s" totalImagesPullingTime="8.225186528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:15.829332927 +0000 UTC m=+11.794315437" watchObservedRunningTime="2026-11-02 14:57:15.829888887 +0000 UTC m=+11.794871399" Nov 02 14:57:15.944792 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:15.944664 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-kjh9p" podStartSLOduration=2.279141668 podStartE2EDuration="10.944644678s" totalImagesPullingTime="8.66550301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:15.944434917 +0000 UTC m=+11.909417426" watchObservedRunningTime="2026-11-02 14:57:15.944644678 +0000 UTC m=+11.909627190" Nov 02 14:57:16.482020 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:16.481767 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:16.482968 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:16.482946 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:16.591788 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:16.591758 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:16.591788 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:16.591789 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:16.591981 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:16.591876 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:16.591981 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:16.591915 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:16.592105 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:16.592030 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:16.592105 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:16.592075 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:16.718159 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:16.718125 2574 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="b41bf56725fab993146a4304c741f421719bd099bedecd52bc950a5be2ff09b7" exitCode=0 Nov 02 14:57:16.718336 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:16.718250 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqtnw" event={"ID":"0444057f-0080-4135-9d83-dca94858033e","Type":"ContainerDied","Data":"b41bf56725fab993146a4304c741f421719bd099bedecd52bc950a5be2ff09b7"} Nov 02 14:57:16.720413 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:16.719542 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:16.720413 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:16.719590 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:57:17.722492 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:17.722450 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xsbm4" event={"ID":"4ffbd00b-29ae-4d63-97b8-edf990224dcd","Type":"ContainerStarted","Data":"8a71dd3f0d96970d76d3921d80129636d10b25ff3161a6ca8edaaca1ee1bd8fd"} Nov 02 14:57:17.724417 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:17.724362 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrz94" event={"ID":"8880c66c-32b2-4a81-a174-34242093784d","Type":"ContainerStarted","Data":"82a37694260643533c61935e34f37db990236cdb26001936d11963c3cf9fb66b"} Nov 02 14:57:17.727312 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:17.727279 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" event={"ID":"b5cc5088-ab54-47e3-bcca-7cb792f88880","Type":"ContainerStarted","Data":"cfc7e9a2fd0bbb6c0c77a2e7f19fb7d778e3de969f994fe146dcd39858006b6e"} Nov 02 14:57:17.837736 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:17.837632 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xsbm4" podStartSLOduration=4.106742397 podStartE2EDuration="12.837616818s" totalImagesPullingTime="8.730874421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:17.837279351 +0000 UTC m=+13.802261860" watchObservedRunningTime="2026-11-02 14:57:17.837616818 +0000 UTC m=+13.802599329" Nov 02 14:57:17.956335 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:17.956274 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" podStartSLOduration=3.2326823239999998 podStartE2EDuration="12.9562544s" totalImagesPullingTime="9.723572076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:17.955816996 +0000 UTC m=+13.920799507" watchObservedRunningTime="2026-11-02 14:57:17.9562544 +0000 UTC m=+13.921236945" Nov 02 14:57:18.074785 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:18.074727 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nrz94" podStartSLOduration=2.472265204 podStartE2EDuration="13.074706798s" totalImagesPullingTime="10.602441594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:18.073773119 +0000 UTC m=+14.038755628" watchObservedRunningTime="2026-11-02 14:57:18.074706798 +0000 UTC m=+14.039689313" Nov 02 14:57:18.588664 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:18.588577 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:18.588664 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:18.588597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:18.588869 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:18.588704 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:18.588869 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:18.588740 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:18.588869 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:18.588813 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:18.589171 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:18.589132 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:18.731286 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:18.731250 2574 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="aa0969f7ada55f6b4530b74257fa5862536791ee04c7777ded259a18a439e50c" exitCode=0 Nov 02 14:57:18.731834 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:18.731346 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqtnw" event={"ID":"0444057f-0080-4135-9d83-dca94858033e","Type":"ContainerDied","Data":"aa0969f7ada55f6b4530b74257fa5862536791ee04c7777ded259a18a439e50c"} Nov 02 14:57:20.589114 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:20.588894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:20.589599 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:20.588922 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:20.589599 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:20.589214 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:20.589599 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:20.588959 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:20.589599 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:20.589320 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:20.589599 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:20.589375 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:20.738371 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:20.738256 2574 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="452828b8f34dbe35232fd3012cff8e4b90097712ecff30ac4bf5bf7522dc7a64" exitCode=0 Nov 02 14:57:20.738371 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:20.738298 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqtnw" event={"ID":"0444057f-0080-4135-9d83-dca94858033e","Type":"ContainerDied","Data":"452828b8f34dbe35232fd3012cff8e4b90097712ecff30ac4bf5bf7522dc7a64"} Nov 02 14:57:21.741657 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:21.741636 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 14:57:21.741986 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:21.741965 2574 generic.go:356] "Generic (PLEG): container finished" podID="449bb762-da7a-4a88-9796-1a1ef23e7c8c" containerID="de2e207e956519c5aa809e15a99d621e932641a49947107f7d6d4a805abe9061" exitCode=1 Nov 02 14:57:21.742052 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:21.742037 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" event={"ID":"449bb762-da7a-4a88-9796-1a1ef23e7c8c","Type":"ContainerStarted","Data":"75ebf9c8999c1360745a72a04924fb6c72c6ea13049601d22a9185ccfdc7a7b4"} Nov 02 14:57:21.742109 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:21.742099 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" event={"ID":"449bb762-da7a-4a88-9796-1a1ef23e7c8c","Type":"ContainerDied","Data":"de2e207e956519c5aa809e15a99d621e932641a49947107f7d6d4a805abe9061"} Nov 02 14:57:21.742137 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:21.742131 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" event={"ID":"449bb762-da7a-4a88-9796-1a1ef23e7c8c","Type":"ContainerStarted","Data":"b9093aa079b3e2e01acf775c13c87990ecde985ad93c3fd190cfa82d410cc036"} Nov 02 14:57:22.297736 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:22.297446 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:22.297860 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:22.297738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:22.297860 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:22.297578 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:22.297860 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:22.297814 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs podName:13823b41-3da3-4a0b-bf1d-4c9d1b24608f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:38.297799313 +0000 UTC m=+34.262781802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs") pod "network-metrics-daemon-f9zqb" (UID: "13823b41-3da3-4a0b-bf1d-4c9d1b24608f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:22.297961 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:22.297888 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:22.297961 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:22.297944 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret podName:4bfe09e4-8fce-4149-a9fb-0a222124f121 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:38.297929048 +0000 UTC m=+34.262911536 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret") pod "global-pull-secret-syncer-vjfjv" (UID: "4bfe09e4-8fce-4149-a9fb-0a222124f121") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:22.399116 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:22.399079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:22.399341 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:22.399277 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:22.399341 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:22.399304 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:22.399341 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:22.399316 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6mjmh for pod openshift-network-diagnostics/network-check-target-vxjb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:22.399469 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:22.399380 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh podName:d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:38.399361345 +0000 UTC m=+34.364343833 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mjmh" (UniqueName: "kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh") pod "network-check-target-vxjb7" (UID: "d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:22.588814 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:22.588609 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:22.588814 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:22.588609 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:22.588814 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:22.588716 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:22.588814 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:22.588734 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:22.589123 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:22.588821 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:22.589123 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:22.588918 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:22.746903 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:22.746880 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 14:57:22.747356 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:22.747261 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" event={"ID":"449bb762-da7a-4a88-9796-1a1ef23e7c8c","Type":"ContainerStarted","Data":"f0df3a682e293044917acaa10e8b06542070ca8369320bf37a9bf0b83824b75c"} Nov 02 14:57:22.747356 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:22.747320 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" event={"ID":"449bb762-da7a-4a88-9796-1a1ef23e7c8c","Type":"ContainerStarted","Data":"22fcabbc63008c9f9b42bfaa56479cad29cb99972b464ef250e87635e43dd7a6"} Nov 02 14:57:22.747356 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:22.747351 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" event={"ID":"449bb762-da7a-4a88-9796-1a1ef23e7c8c","Type":"ContainerStarted","Data":"ca18ed14a9db94c8f63cee90150ea88d7c1988603ea9ea49a177350ab976e972"} Nov 02 14:57:23.751533 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:23.751494 2574 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="24cee5b3b619e6260fe749e973764c3a09397404befb46177fcb7411fddf97c1" exitCode=0 Nov 02 14:57:23.752121 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:23.751555 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqtnw" event={"ID":"0444057f-0080-4135-9d83-dca94858033e","Type":"ContainerDied","Data":"24cee5b3b619e6260fe749e973764c3a09397404befb46177fcb7411fddf97c1"} Nov 02 14:57:24.589664 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:24.589617 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:24.589879 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:24.589730 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:24.589879 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:24.589747 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:24.589879 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:24.589819 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:24.589879 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:24.589853 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:24.590090 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:24.589918 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:24.756147 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:24.756118 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 14:57:24.756725 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:24.756514 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" event={"ID":"449bb762-da7a-4a88-9796-1a1ef23e7c8c","Type":"ContainerStarted","Data":"be12346687fd559c45923bcdc9c206edc3468691bcf6559b9c8cbd8e0dfc15bc"} Nov 02 14:57:24.759433 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:24.759407 2574 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="52a3e3500c303e360343d669230281a4591cc1024cbd6a0aec2e047d9b59543c" exitCode=0 Nov 02 14:57:24.759548 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:24.759433 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqtnw" event={"ID":"0444057f-0080-4135-9d83-dca94858033e","Type":"ContainerDied","Data":"52a3e3500c303e360343d669230281a4591cc1024cbd6a0aec2e047d9b59543c"} Nov 02 14:57:25.764873 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:25.764842 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lqtnw" event={"ID":"0444057f-0080-4135-9d83-dca94858033e","Type":"ContainerStarted","Data":"301eb1b00e9ee38d2ff71eb52712c58d87e723e11576e6c504e641fdf58ae620"} Nov 02 14:57:25.893333 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:25.893286 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lqtnw" podStartSLOduration=7.873424973 podStartE2EDuration="20.893272976s" totalImagesPullingTime="13.019848003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:25.892843989 +0000 UTC m=+21.857826500" watchObservedRunningTime="2026-11-02 14:57:25.893272976 +0000 UTC m=+21.858255486" Nov 02 14:57:26.589134 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:26.588800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:26.589134 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:26.588852 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:26.589316 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:26.589164 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:26.589316 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:26.588916 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:26.589316 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:26.589207 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:26.589316 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:26.589269 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:26.769777 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:26.769749 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 14:57:26.770330 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:26.770152 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" event={"ID":"449bb762-da7a-4a88-9796-1a1ef23e7c8c","Type":"ContainerStarted","Data":"4a72f8245a6dd5f5b4ed104014bfe54aa9e63591c0239b6bcfc8f1e6dd421df4"} Nov 02 14:57:26.770726 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:26.770709 2574 scope.go:122] "RemoveContainer" containerID="de2e207e956519c5aa809e15a99d621e932641a49947107f7d6d4a805abe9061" Nov 02 14:57:26.791296 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:26.791277 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kjh9p_b3d8b64c-559c-4806-8599-25b62c630fb4/dns-node-resolver/0.log" Nov 02 14:57:27.775939 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:27.775910 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 14:57:27.776361 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:27.776269 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" event={"ID":"449bb762-da7a-4a88-9796-1a1ef23e7c8c","Type":"ContainerStarted","Data":"e49f15a4f73ece70e22a1ad674427f22131a7c9c5e6082399831ec54a21f271b"} Nov 02 14:57:27.776642 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:27.776617 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:27.776758 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:27.776685 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:27.776758 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:27.776733 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:27.791029 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:27.790987 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:27.792130 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:27.792117 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:57:27.824135 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:27.824114 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zdkx8_2e7e4e7b-c190-420d-b438-b2c9e1074795/node-ca/0.log" Nov 02 14:57:27.968060 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:27.967988 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" podStartSLOduration=8.429586122 podStartE2EDuration="22.967974702s" totalImagesPullingTime="14.53838858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:27.967652328 +0000 UTC m=+23.932634838" watchObservedRunningTime="2026-11-02 14:57:27.967974702 +0000 UTC m=+23.932957212" Nov 02 14:57:28.589068 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:28.589039 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:28.589294 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:28.589092 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:28.589294 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:28.589096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:28.589294 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:28.589164 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:28.589294 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:28.589245 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:28.589440 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:28.589310 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:30.591100 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:30.591074 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:30.591496 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:30.591074 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:30.591496 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:30.591175 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:30.591496 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:30.591257 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:30.591496 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:30.591078 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:30.591496 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:30.591320 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:32.591908 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:32.591880 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:32.592425 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:32.591980 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:32.592425 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:32.591987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:32.592425 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:32.592094 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:32.592425 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:32.592119 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:32.592425 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:32.592179 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:34.589300 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:34.589265 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:34.589699 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:34.589360 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:34.589699 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:34.589451 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:34.589699 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:34.589542 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:34.589699 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:34.589593 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:34.589699 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:34.589663 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:36.593238 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:36.593207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:36.593599 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:36.593279 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:36.593599 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:36.593303 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:36.593599 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:36.593375 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:36.593599 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:36.593412 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:36.593599 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:36.593427 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:38.308594 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:38.308565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:38.309019 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:38.308622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:38.309019 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:38.308710 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:38.309019 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:38.308711 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:38.309019 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:38.308756 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs podName:13823b41-3da3-4a0b-bf1d-4c9d1b24608f nodeName:}" failed. No retries permitted until 2026-11-02 14:58:10.308743144 +0000 UTC m=+66.273725645 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs") pod "network-metrics-daemon-f9zqb" (UID: "13823b41-3da3-4a0b-bf1d-4c9d1b24608f") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:38.309019 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:38.308768 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret podName:4bfe09e4-8fce-4149-a9fb-0a222124f121 nodeName:}" failed. No retries permitted until 2026-11-02 14:58:10.308762568 +0000 UTC m=+66.273745056 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret") pod "global-pull-secret-syncer-vjfjv" (UID: "4bfe09e4-8fce-4149-a9fb-0a222124f121") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:38.408862 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:38.408832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:38.409029 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:38.408943 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:38.409029 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:38.408955 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:38.409029 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:38.408964 2574 projected.go:196] Error preparing data for projected volume kube-api-access-6mjmh for pod openshift-network-diagnostics/network-check-target-vxjb7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:38.409029 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:38.409014 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh podName:d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9 nodeName:}" failed. No retries permitted until 2026-11-02 14:58:10.40900351 +0000 UTC m=+66.373986011 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6mjmh" (UniqueName: "kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh") pod "network-check-target-vxjb7" (UID: "d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:38.591057 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:38.590975 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:38.591217 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:38.590975 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:38.591217 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:38.591078 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:38.591217 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:38.591170 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:38.591217 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:38.590975 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:38.591360 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:38.591271 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:40.588185 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:40.588155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:40.588185 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:40.588168 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:40.588599 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:40.588155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:40.588599 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:40.588261 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:40.588599 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:40.588342 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:40.588599 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:40.588403 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:42.588883 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:42.588852 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:42.589265 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:42.588945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:42.589265 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:42.588949 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:42.589265 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:42.589065 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:42.589265 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:42.589116 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:42.589265 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:42.589171 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:44.590889 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:44.590862 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:44.591428 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:44.590958 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:44.591428 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:44.591074 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:44.591428 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:44.591109 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:44.591428 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:44.591103 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:44.591428 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:44.591164 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:46.588817 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:46.588648 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:46.589212 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:46.588648 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:46.589212 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:46.588899 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:46.589212 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:46.588669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:46.589212 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:46.588970 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:46.589212 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:46.589071 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:46.900338 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:46.900309 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vjfjv"] Nov 02 14:57:46.900470 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:46.900434 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:46.900573 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:46.900548 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:46.903119 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:46.903097 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vxjb7"] Nov 02 14:57:46.903215 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:46.903203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:46.903317 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:46.903300 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:46.903596 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:46.903578 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f9zqb"] Nov 02 14:57:46.903675 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:46.903663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:46.903780 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:46.903762 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:48.588871 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:48.588787 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:48.588871 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:48.588824 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:48.589512 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:48.588793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:48.589512 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:48.588930 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:48.589512 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:48.588989 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:48.589512 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:48.589080 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:50.588952 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:50.588915 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:50.589485 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:50.588922 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:50.589485 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:50.589059 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f9zqb" podUID="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" Nov 02 14:57:50.589485 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:50.589070 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:50.589485 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:50.589144 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vxjb7" podUID="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" Nov 02 14:57:50.589485 ip-10-0-142-141 kubenswrapper[2574]: E1102 14:57:50.589209 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vjfjv" podUID="4bfe09e4-8fce-4149-a9fb-0a222124f121" Nov 02 14:57:51.285089 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.285059 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeReady" Nov 02 14:57:51.285247 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.285186 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 14:57:51.452012 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.451977 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qlm5f"] Nov 02 14:57:51.460591 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.460564 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lgnjt"] Nov 02 14:57:51.460744 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.460685 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.463040 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.463017 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vqgwt"] Nov 02 14:57:51.464669 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.464649 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.475614 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.475596 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lgnjt"] Nov 02 14:57:51.475700 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.475634 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qlm5f"] Nov 02 14:57:51.475700 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.475663 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vqgwt"] Nov 02 14:57:51.475761 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.475724 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:57:51.515290 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.515264 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 14:57:51.515545 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.515530 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cr8rg\"" Nov 02 14:57:51.515595 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.515557 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 14:57:51.518095 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.518083 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 14:57:51.555393 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.555346 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 14:57:51.555601 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.555570 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c2bv5\"" Nov 02 14:57:51.555813 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.555799 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 14:57:51.555851 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.555809 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 14:57:51.555883 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.555859 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 14:57:51.564747 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.564728 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4d7td\"" Nov 02 14:57:51.564804 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.564756 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 14:57:51.564804 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.564757 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 14:57:51.601183 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.601144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25f92d33-6475-49f1-bfff-691a5dec0ae0-data-volume\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.601183 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.601184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989be356-90d5-4c62-8200-5d5a8d2af857-tmp-dir\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.601659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.601200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsfph\" (UniqueName: \"kubernetes.io/projected/989be356-90d5-4c62-8200-5d5a8d2af857-kube-api-access-jsfph\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.601659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.601308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.601659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.601353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25f92d33-6475-49f1-bfff-691a5dec0ae0-crio-socket\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.601659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.601368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/989be356-90d5-4c62-8200-5d5a8d2af857-config-volume\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.601659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.601394 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25f92d33-6475-49f1-bfff-691a5dec0ae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.601659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.601409 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66740b77-b82e-49c2-b8d0-af48d25147cb-cert\") pod \"ingress-canary-vqgwt\" (UID: \"66740b77-b82e-49c2-b8d0-af48d25147cb\") " pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:57:51.601659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.601507 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/989be356-90d5-4c62-8200-5d5a8d2af857-metrics-tls\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.601659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.601532 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2slcd\" (UniqueName: \"kubernetes.io/projected/66740b77-b82e-49c2-b8d0-af48d25147cb-kube-api-access-2slcd\") pod \"ingress-canary-vqgwt\" (UID: \"66740b77-b82e-49c2-b8d0-af48d25147cb\") " pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:57:51.601659 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.601555 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcklx\" (UniqueName: \"kubernetes.io/projected/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-api-access-wcklx\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.702633 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.702599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/989be356-90d5-4c62-8200-5d5a8d2af857-metrics-tls\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.702633 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.702632 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2slcd\" (UniqueName: \"kubernetes.io/projected/66740b77-b82e-49c2-b8d0-af48d25147cb-kube-api-access-2slcd\") pod \"ingress-canary-vqgwt\" (UID: \"66740b77-b82e-49c2-b8d0-af48d25147cb\") " pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:57:51.702822 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.702655 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcklx\" (UniqueName: \"kubernetes.io/projected/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-api-access-wcklx\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.702822 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.702778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25f92d33-6475-49f1-bfff-691a5dec0ae0-data-volume\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.702822 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.702812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989be356-90d5-4c62-8200-5d5a8d2af857-tmp-dir\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.702964 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.702833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsfph\" (UniqueName: \"kubernetes.io/projected/989be356-90d5-4c62-8200-5d5a8d2af857-kube-api-access-jsfph\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.702964 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.702875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.702964 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.702909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25f92d33-6475-49f1-bfff-691a5dec0ae0-crio-socket\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.702964 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.702933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/989be356-90d5-4c62-8200-5d5a8d2af857-config-volume\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.703190 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.702974 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25f92d33-6475-49f1-bfff-691a5dec0ae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.703190 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.703016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66740b77-b82e-49c2-b8d0-af48d25147cb-cert\") pod \"ingress-canary-vqgwt\" (UID: \"66740b77-b82e-49c2-b8d0-af48d25147cb\") " pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:57:51.703287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.703213 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989be356-90d5-4c62-8200-5d5a8d2af857-tmp-dir\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.703287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.703224 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25f92d33-6475-49f1-bfff-691a5dec0ae0-crio-socket\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.703287 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.703213 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25f92d33-6475-49f1-bfff-691a5dec0ae0-data-volume\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.703584 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.703558 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.703673 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.703563 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/989be356-90d5-4c62-8200-5d5a8d2af857-config-volume\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.706602 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.706577 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/989be356-90d5-4c62-8200-5d5a8d2af857-metrics-tls\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.706707 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.706648 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25f92d33-6475-49f1-bfff-691a5dec0ae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.706707 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.706679 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66740b77-b82e-49c2-b8d0-af48d25147cb-cert\") pod \"ingress-canary-vqgwt\" (UID: \"66740b77-b82e-49c2-b8d0-af48d25147cb\") " pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:57:51.764045 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.764014 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2slcd\" (UniqueName: \"kubernetes.io/projected/66740b77-b82e-49c2-b8d0-af48d25147cb-kube-api-access-2slcd\") pod \"ingress-canary-vqgwt\" (UID: \"66740b77-b82e-49c2-b8d0-af48d25147cb\") " pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:57:51.764195 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.764013 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsfph\" (UniqueName: \"kubernetes.io/projected/989be356-90d5-4c62-8200-5d5a8d2af857-kube-api-access-jsfph\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.764195 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.764015 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcklx\" (UniqueName: \"kubernetes.io/projected/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-api-access-wcklx\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:51.792418 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.792386 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:51.808676 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.808628 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:57:51.812410 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:51.812386 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:57:52.246227 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.246196 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qlm5f"] Nov 02 14:57:52.249826 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:52.249797 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989be356_90d5_4c62_8200_5d5a8d2af857.slice/crio-cf603277525bbb9571d2fdcc94ee1ec1f423ca8841deea710e422425860fbd44 WatchSource:0}: Error finding container cf603277525bbb9571d2fdcc94ee1ec1f423ca8841deea710e422425860fbd44: Status 404 returned error can't find the container with id cf603277525bbb9571d2fdcc94ee1ec1f423ca8841deea710e422425860fbd44 Nov 02 14:57:52.279988 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.279969 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vqgwt"] Nov 02 14:57:52.282042 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:52.282018 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66740b77_b82e_49c2_b8d0_af48d25147cb.slice/crio-5aedfa6ce17b0d33e1957bf4174f6e175ce936cab0557a8516b5e02adeec7d71 WatchSource:0}: Error finding container 5aedfa6ce17b0d33e1957bf4174f6e175ce936cab0557a8516b5e02adeec7d71: Status 404 returned error can't find the container with id 5aedfa6ce17b0d33e1957bf4174f6e175ce936cab0557a8516b5e02adeec7d71 Nov 02 14:57:52.282486 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.282470 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lgnjt"] Nov 02 14:57:52.298878 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:52.298857 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f92d33_6475_49f1_bfff_691a5dec0ae0.slice/crio-ae112bf9afbf161f0542ba34faad93de100d37b0ff254d7d17bacc6286ce8be0 WatchSource:0}: Error finding container ae112bf9afbf161f0542ba34faad93de100d37b0ff254d7d17bacc6286ce8be0: Status 404 returned error can't find the container with id ae112bf9afbf161f0542ba34faad93de100d37b0ff254d7d17bacc6286ce8be0 Nov 02 14:57:52.588866 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.588776 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:57:52.588866 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.588805 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:57:52.589079 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.589015 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:57:52.643243 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.643215 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gmvcm\"" Nov 02 14:57:52.643243 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.643215 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-29n45\"" Nov 02 14:57:52.643609 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.643266 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gwmlb\"" Nov 02 14:57:52.643609 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.643343 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 14:57:52.644929 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.644916 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 14:57:52.644970 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.644927 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 14:57:52.646612 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.646597 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 14:57:52.825975 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.825943 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vqgwt" event={"ID":"66740b77-b82e-49c2-b8d0-af48d25147cb","Type":"ContainerStarted","Data":"5aedfa6ce17b0d33e1957bf4174f6e175ce936cab0557a8516b5e02adeec7d71"} Nov 02 14:57:52.827085 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.827057 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qlm5f" event={"ID":"989be356-90d5-4c62-8200-5d5a8d2af857","Type":"ContainerStarted","Data":"cf603277525bbb9571d2fdcc94ee1ec1f423ca8841deea710e422425860fbd44"} Nov 02 14:57:52.828359 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.828333 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lgnjt" event={"ID":"25f92d33-6475-49f1-bfff-691a5dec0ae0","Type":"ContainerStarted","Data":"204584f8caa870d4293234fc18b0512aee988295ff87abbf7e0d2209c12d438a"} Nov 02 14:57:52.828451 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:52.828377 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lgnjt" event={"ID":"25f92d33-6475-49f1-bfff-691a5dec0ae0","Type":"ContainerStarted","Data":"ae112bf9afbf161f0542ba34faad93de100d37b0ff254d7d17bacc6286ce8be0"} Nov 02 14:57:53.578004 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.577969 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kwcld"] Nov 02 14:57:53.581115 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.581081 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.687371 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.687333 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 14:57:53.688507 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.688484 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kwcld"] Nov 02 14:57:53.688741 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.688719 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 14:57:53.688969 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.688952 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 14:57:53.689082 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.689022 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 14:57:53.689143 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.689136 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 14:57:53.716989 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.716922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.717121 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.717012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.717121 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.717047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1af9240-8442-4911-b1d9-5570aea28abf-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.717121 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.717066 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smgsz\" (UniqueName: \"kubernetes.io/projected/d1af9240-8442-4911-b1d9-5570aea28abf-kube-api-access-smgsz\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.737609 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.737576 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l9qfj\"" Nov 02 14:57:53.817964 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.817927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.818133 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.817984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1af9240-8442-4911-b1d9-5570aea28abf-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.818133 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.818025 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smgsz\" (UniqueName: \"kubernetes.io/projected/d1af9240-8442-4911-b1d9-5570aea28abf-kube-api-access-smgsz\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.818133 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.818068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.818788 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.818761 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1af9240-8442-4911-b1d9-5570aea28abf-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.820859 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.820830 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.820961 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.820841 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.832887 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.832828 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lgnjt" event={"ID":"25f92d33-6475-49f1-bfff-691a5dec0ae0","Type":"ContainerStarted","Data":"2d5ac321445f4d0e0dd2a914daf56519650d410fd4c211f82a15cc8f2fb5457e"} Nov 02 14:57:53.880874 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.880846 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smgsz\" (UniqueName: \"kubernetes.io/projected/d1af9240-8442-4911-b1d9-5570aea28abf-kube-api-access-smgsz\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:53.901660 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:53.901637 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:57:54.279132 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:54.279095 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kwcld"] Nov 02 14:57:54.283193 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:57:54.283165 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1af9240_8442_4911_b1d9_5570aea28abf.slice/crio-9932e1dc75fd99bb713c16465e83cf2f113d3fcfea5ae912e13fa8c42434a2e9 WatchSource:0}: Error finding container 9932e1dc75fd99bb713c16465e83cf2f113d3fcfea5ae912e13fa8c42434a2e9: Status 404 returned error can't find the container with id 9932e1dc75fd99bb713c16465e83cf2f113d3fcfea5ae912e13fa8c42434a2e9 Nov 02 14:57:54.838417 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:54.838330 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qlm5f" event={"ID":"989be356-90d5-4c62-8200-5d5a8d2af857","Type":"ContainerStarted","Data":"f05636607d8b477bd94e64a789c23e7cd9983c20f2e11ebbbc1aeb9a7fd9c944"} Nov 02 14:57:54.838417 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:54.838398 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qlm5f" event={"ID":"989be356-90d5-4c62-8200-5d5a8d2af857","Type":"ContainerStarted","Data":"ff79a837fa0e0b357a277dd711573d3c486e11ae90aedd4ef810ebefcd52fb8d"} Nov 02 14:57:54.838896 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:54.838448 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qlm5f" Nov 02 14:57:54.839598 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:54.839575 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" event={"ID":"d1af9240-8442-4911-b1d9-5570aea28abf","Type":"ContainerStarted","Data":"9932e1dc75fd99bb713c16465e83cf2f113d3fcfea5ae912e13fa8c42434a2e9"} Nov 02 14:57:54.973776 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:54.973712 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-qlm5f" podStartSLOduration=1.656335844 podStartE2EDuration="3.973693211s" totalImagesPullingTime="2.317357367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:54.972808821 +0000 UTC m=+50.937791333" watchObservedRunningTime="2026-11-02 14:57:54.973693211 +0000 UTC m=+50.938675723" Nov 02 14:57:56.845542 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:56.845507 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vqgwt" event={"ID":"66740b77-b82e-49c2-b8d0-af48d25147cb","Type":"ContainerStarted","Data":"2e9b443e6afeccffaee8c28ecc3c502639597def622ea3b76137684a34b8803f"} Nov 02 14:57:56.847276 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:56.847241 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" event={"ID":"d1af9240-8442-4911-b1d9-5570aea28abf","Type":"ContainerStarted","Data":"cf50850b82e9e3314e22f7913f57eafad32eceb45184d752e5ff0c0438ea94f2"} Nov 02 14:57:56.847404 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:56.847287 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" event={"ID":"d1af9240-8442-4911-b1d9-5570aea28abf","Type":"ContainerStarted","Data":"83ca0bf65cd680469e9ad7e327dd4f5f2858dc6ebeea44e2cb15bc946ee53b52"} Nov 02 14:57:56.849079 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:56.849056 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lgnjt" event={"ID":"25f92d33-6475-49f1-bfff-691a5dec0ae0","Type":"ContainerStarted","Data":"1dc41dcddac08074df46583880a9aa0a5bae2ac1270af0eb12e020dc05a28113"} Nov 02 14:57:56.964297 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:56.964249 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vqgwt" podStartSLOduration=1.945739787 podStartE2EDuration="5.964236906s" totalImagesPullingTime="4.018497119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:56.963685355 +0000 UTC m=+52.928667865" watchObservedRunningTime="2026-11-02 14:57:56.964236906 +0000 UTC m=+52.929219417" Nov 02 14:57:57.082597 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:57.082543 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" podStartSLOduration=2.115965344 podStartE2EDuration="4.082516707s" totalImagesPullingTime="1.966551363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:57.082233086 +0000 UTC m=+53.047215597" watchObservedRunningTime="2026-11-02 14:57:57.082516707 +0000 UTC m=+53.047499219" Nov 02 14:57:57.203090 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:57.203034 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lgnjt" podStartSLOduration=2.7916922360000003 podStartE2EDuration="6.203020489s" totalImagesPullingTime="3.411328253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:57:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:57.202369917 +0000 UTC m=+53.167352427" watchObservedRunningTime="2026-11-02 14:57:57.203020489 +0000 UTC m=+53.168002991" Nov 02 14:57:59.791209 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:57:59.791180 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:00.272270 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.272235 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qzm49"] Nov 02 14:58:00.275592 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.275576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.371436 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.371408 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 14:58:00.374595 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.374575 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 14:58:00.374595 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.374590 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zw4g4\"" Nov 02 14:58:00.374731 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.374577 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 14:58:00.471120 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.471087 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-tls\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.471120 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.471122 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-textfile\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.471362 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.471145 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-metrics-client-ca\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.471362 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.471175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-wtmp\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.471362 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.471225 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz25j\" (UniqueName: \"kubernetes.io/projected/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-kube-api-access-hz25j\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.471362 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.471285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-sys\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.471362 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.471328 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-accelerators-collector-config\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.471520 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.471400 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-root\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.471520 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.471486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572311 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-tls\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572311 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-textfile\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572311 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-metrics-client-ca\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572311 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-wtmp\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572646 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz25j\" (UniqueName: \"kubernetes.io/projected/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-kube-api-access-hz25j\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572646 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-sys\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572646 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-accelerators-collector-config\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572646 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-root\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572646 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572433 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-sys\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572646 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572448 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572646 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572471 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-wtmp\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.572646 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572533 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-root\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.573017 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572657 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-textfile\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.573017 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572923 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-metrics-client-ca\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.573017 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.572970 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-accelerators-collector-config\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.574624 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.574603 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-tls\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.574711 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.574648 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.642345 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.642314 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz25j\" (UniqueName: \"kubernetes.io/projected/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-kube-api-access-hz25j\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.890404 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:00.890369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:00.900331 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:58:00.900299 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd186e7b3_d2fa_4f5b_bd31_ee633608c8db.slice/crio-f6a70d6e7a0d42f7702cdb67e0bdfe3cbf436f17b1c762ac22bbd768071cc2ce WatchSource:0}: Error finding container f6a70d6e7a0d42f7702cdb67e0bdfe3cbf436f17b1c762ac22bbd768071cc2ce: Status 404 returned error can't find the container with id f6a70d6e7a0d42f7702cdb67e0bdfe3cbf436f17b1c762ac22bbd768071cc2ce Nov 02 14:58:01.868009 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:01.867959 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qzm49" event={"ID":"d186e7b3-d2fa-4f5b-bd31-ee633608c8db","Type":"ContainerStarted","Data":"f6a70d6e7a0d42f7702cdb67e0bdfe3cbf436f17b1c762ac22bbd768071cc2ce"} Nov 02 14:58:02.658497 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.656399 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 14:58:02.662894 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.662869 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.686619 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.686590 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.686619 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.686621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.686793 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.686640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.686793 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.686666 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.686793 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.686737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.686888 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.686855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.686888 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.686884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.686973 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.686915 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.687096 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.687020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.687096 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.687068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-volume\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.687178 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.687107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-out\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.687178 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.687163 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zjwc\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-kube-api-access-8zjwc\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.687236 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.687209 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-web-config\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.718187 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.718146 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 14:58:02.788097 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788097 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788326 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788326 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788326 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788183 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-volume\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788326 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-out\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788326 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zjwc\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-kube-api-access-8zjwc\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788582 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-web-config\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788582 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788582 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788582 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788582 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788517 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788582 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788544 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788582 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788568 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.788898 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.788880 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.790414 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.790397 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-out\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.819550 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.819460 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 14:58:02.819550 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.819464 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-89ttf\"" Nov 02 14:58:02.819550 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.819464 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 14:58:02.819550 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.819523 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 14:58:02.819550 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.819531 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 14:58:02.819550 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.819471 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 14:58:02.819873 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.819602 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 14:58:02.819873 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.819620 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 14:58:02.819873 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.819684 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 14:58:02.819873 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.819824 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 14:58:02.830032 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.829953 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.832156 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.832132 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.832156 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.832145 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.832416 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.832145 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.832416 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.832188 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.832416 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.832188 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-web-config\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.832602 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.832528 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.832602 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.832521 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.833841 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.833822 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-volume\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.850375 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.850352 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zjwc\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-kube-api-access-8zjwc\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:02.872083 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.872010 2574 generic.go:356] "Generic (PLEG): container finished" podID="d186e7b3-d2fa-4f5b-bd31-ee633608c8db" containerID="37e5f1844a97c4c134d019273ae6d0494d3523081bef1aedc7bc712b18b82c85" exitCode=0 Nov 02 14:58:02.872083 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.872044 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qzm49" event={"ID":"d186e7b3-d2fa-4f5b-bd31-ee633608c8db","Type":"ContainerDied","Data":"37e5f1844a97c4c134d019273ae6d0494d3523081bef1aedc7bc712b18b82c85"} Nov 02 14:58:02.987018 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:02.986983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:03.434360 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:03.434324 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 14:58:03.437295 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:58:03.437267 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b78c94_8df4_49a1_b365_bc0b92eae8ba.slice/crio-5015acc47cf4b5020f7b0f0e4faf6d8dc1c63d0b7daa01fa7708869ebc020ac6 WatchSource:0}: Error finding container 5015acc47cf4b5020f7b0f0e4faf6d8dc1c63d0b7daa01fa7708869ebc020ac6: Status 404 returned error can't find the container with id 5015acc47cf4b5020f7b0f0e4faf6d8dc1c63d0b7daa01fa7708869ebc020ac6 Nov 02 14:58:03.879118 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:03.879074 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qzm49" event={"ID":"d186e7b3-d2fa-4f5b-bd31-ee633608c8db","Type":"ContainerStarted","Data":"eec6c93beaa70858601fd827086f7528698237e03166568e5b686ce0d33a6142"} Nov 02 14:58:03.879607 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:03.879156 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qzm49" event={"ID":"d186e7b3-d2fa-4f5b-bd31-ee633608c8db","Type":"ContainerStarted","Data":"c4c09bf5edc92b1739387d1251ecc06c6103ad299c87c1ab04f6f5b541f69e16"} Nov 02 14:58:03.880580 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:03.880556 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerStarted","Data":"5015acc47cf4b5020f7b0f0e4faf6d8dc1c63d0b7daa01fa7708869ebc020ac6"} Nov 02 14:58:04.004060 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:04.003988 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qzm49" podStartSLOduration=2.217460655 podStartE2EDuration="4.003974263s" totalImagesPullingTime="1.716732221s" totalInitContainerRuntime="69.781387ms" isStatefulPod=true podCreationTimestamp="2026-11-02 14:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:04.003495265 +0000 UTC m=+59.968477778" watchObservedRunningTime="2026-11-02 14:58:04.003974263 +0000 UTC m=+59.968956776" Nov 02 14:58:04.844719 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:04.844686 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qlm5f" Nov 02 14:58:05.892266 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:05.889190 2574 generic.go:356] "Generic (PLEG): container finished" podID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerID="8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9" exitCode=0 Nov 02 14:58:05.892266 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:05.889262 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerDied","Data":"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9"} Nov 02 14:58:06.125103 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.125070 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7899fb9795-f778g"] Nov 02 14:58:06.128328 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.128312 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.182580 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.182491 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 14:58:06.182958 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.182933 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 14:58:06.183276 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.183257 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 14:58:06.183432 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.183348 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 14:58:06.183753 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.183736 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 14:58:06.210425 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.210392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.210567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.210453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.210567 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.210537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-serving-certs-ca-bundle\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.210666 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.210570 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-client-tls\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.210666 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.210590 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmc4m\" (UniqueName: \"kubernetes.io/projected/5e8592fc-f083-4eb8-b934-517c47f123c4-kube-api-access-fmc4m\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.210666 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.210637 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.210762 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.210741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-metrics-client-ca\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.210791 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.210770 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-federate-client-tls\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.232921 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.232895 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rxzk9\"" Nov 02 14:58:06.233926 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.233909 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 14:58:06.234609 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.234588 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7899fb9795-f778g"] Nov 02 14:58:06.311978 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.311943 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.312159 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.312019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.312159 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.312048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-serving-certs-ca-bundle\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.312159 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.312073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-client-tls\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.312159 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.312098 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmc4m\" (UniqueName: \"kubernetes.io/projected/5e8592fc-f083-4eb8-b934-517c47f123c4-kube-api-access-fmc4m\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.312305 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.312245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.312305 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.312293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-metrics-client-ca\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.312417 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.312336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-federate-client-tls\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.313034 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.313007 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-metrics-client-ca\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.313142 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.313057 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-serving-certs-ca-bundle\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.313235 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.313213 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.314724 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.314702 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.314855 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.314837 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-federate-client-tls\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.315077 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.315061 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.315157 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.315095 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-client-tls\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.373869 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.373834 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmc4m\" (UniqueName: \"kubernetes.io/projected/5e8592fc-f083-4eb8-b934-517c47f123c4-kube-api-access-fmc4m\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.443624 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.443530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:06.872871 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.872838 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7899fb9795-f778g"] Nov 02 14:58:06.875599 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:58:06.875568 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8592fc_f083_4eb8_b934_517c47f123c4.slice/crio-b0e7b5b1f713c379c6ad72e211f011a4861cea6c16607f5c8be4f71aecfbb0d6 WatchSource:0}: Error finding container b0e7b5b1f713c379c6ad72e211f011a4861cea6c16607f5c8be4f71aecfbb0d6: Status 404 returned error can't find the container with id b0e7b5b1f713c379c6ad72e211f011a4861cea6c16607f5c8be4f71aecfbb0d6 Nov 02 14:58:06.893256 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:06.893228 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" event={"ID":"5e8592fc-f083-4eb8-b934-517c47f123c4","Type":"ContainerStarted","Data":"b0e7b5b1f713c379c6ad72e211f011a4861cea6c16607f5c8be4f71aecfbb0d6"} Nov 02 14:58:07.897578 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:07.897545 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerStarted","Data":"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497"} Nov 02 14:58:08.905971 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:08.905928 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerStarted","Data":"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841"} Nov 02 14:58:08.906382 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:08.906034 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerStarted","Data":"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa"} Nov 02 14:58:08.906382 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:08.906080 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerStarted","Data":"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4"} Nov 02 14:58:08.906382 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:08.906117 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerStarted","Data":"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a"} Nov 02 14:58:09.813912 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.813834 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:09.817425 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.817406 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842266 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842225 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842266 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-config\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842437 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842489 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842431 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842489 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842462 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842489 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842485 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842570 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842550 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842599 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842631 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842662 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842694 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842675 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842724 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842753 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842786 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842765 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842814 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842844 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842844 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842900 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.842900 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxp4\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-kube-api-access-vsxp4\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.843009 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.842985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.864008 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.863969 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:09.883349 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.883317 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 14:58:09.883349 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.883330 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 14:58:09.883504 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.883357 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 14:58:09.886409 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.886387 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 14:58:09.913258 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.913223 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerStarted","Data":"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b"} Nov 02 14:58:09.915542 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.915513 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" event={"ID":"5e8592fc-f083-4eb8-b934-517c47f123c4","Type":"ContainerStarted","Data":"e019781104c6500f383aa66617e26d438fd391eefdf0f41bee88ac0ccb986515"} Nov 02 14:58:09.915640 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.915561 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" event={"ID":"5e8592fc-f083-4eb8-b934-517c47f123c4","Type":"ContainerStarted","Data":"8ca53df42b9701a628a2a214dfc5215acb34779d30be754f40e4e84c707c0500"} Nov 02 14:58:09.915640 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.915582 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" event={"ID":"5e8592fc-f083-4eb8-b934-517c47f123c4","Type":"ContainerStarted","Data":"0fd3e4ac548e2c8cbe07e95c724ead8e2f239ae7ee2fad6bd0072b2802eb0dbe"} Nov 02 14:58:09.943361 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.943332 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.943361 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.943361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.943611 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.943393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.943611 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.943417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.943611 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.943440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.943611 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.943575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxp4\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-kube-api-access-vsxp4\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.943814 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.943668 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.943814 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.943727 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.943814 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.943752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-config\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.943814 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.943775 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944073 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944054 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944073 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944061 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944178 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944157 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944238 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944207 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944298 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944346 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944393 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944448 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944505 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944469 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944505 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944493 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944610 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944670 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944755 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944734 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.944810 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.944777 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.945290 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.945266 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.945485 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.945470 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.946053 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.946037 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.968319 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.968296 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 14:58:09.976549 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.976529 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.991484 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991346 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 14:58:09.991484 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991384 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 14:58:09.991484 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991411 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 14:58:09.991484 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991429 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cp3c23k1g144d\"" Nov 02 14:58:09.991484 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991444 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 14:58:09.991484 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991453 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-89xs6\"" Nov 02 14:58:09.991484 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991457 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 14:58:09.991484 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991445 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 14:58:09.991484 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991477 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 14:58:09.991829 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991520 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 14:58:09.991829 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991539 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 14:58:09.991829 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.991570 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 14:58:09.996198 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.996128 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.997695 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.996635 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.997695 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.996789 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.997695 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.996804 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.997695 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.996852 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-config\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.997695 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.997160 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.997695 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.997627 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.999847 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.999825 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.999933 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.999887 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:09.999933 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.999885 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:10.000044 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:09.999940 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:10.020608 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.020581 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxp4\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-kube-api-access-vsxp4\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:10.125166 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.125118 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.228061146 podStartE2EDuration="8.125104727s" totalImagesPullingTime="4.888184501s" totalInitContainerRuntime="8.85908ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:10.124972085 +0000 UTC m=+66.089954597" watchObservedRunningTime="2026-11-02 14:58:10.125104727 +0000 UTC m=+66.090087239" Nov 02 14:58:10.141225 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.141196 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:10.288938 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.288882 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" podStartSLOduration=1.959192499 podStartE2EDuration="4.288870095s" totalImagesPullingTime="2.329677596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:10.287894678 +0000 UTC m=+66.252877188" watchObservedRunningTime="2026-11-02 14:58:10.288870095 +0000 UTC m=+66.253852606" Nov 02 14:58:10.348448 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.348413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:58:10.348566 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.348488 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:58:10.400482 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.400421 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 14:58:10.400599 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.400497 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 14:58:10.411706 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.411688 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:58:10.411779 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.411715 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:58:10.448877 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.448844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:58:10.501252 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.501226 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 14:58:10.561745 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.561719 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 14:58:10.572257 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.572226 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:58:10.579476 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.579454 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:10.582087 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:58:10.582065 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddab9016_d633_455c_a5e4_477bad67508d.slice/crio-b7d549e975284753735ba54b893e6a085455f508ae6ee948a321e2b5c09e20da WatchSource:0}: Error finding container b7d549e975284753735ba54b893e6a085455f508ae6ee948a321e2b5c09e20da: Status 404 returned error can't find the container with id b7d549e975284753735ba54b893e6a085455f508ae6ee948a321e2b5c09e20da Nov 02 14:58:10.655187 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.655043 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gmvcm\"" Nov 02 14:58:10.658308 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.658284 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gwmlb\"" Nov 02 14:58:10.663488 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.663467 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:58:10.666123 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.666103 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-29n45\"" Nov 02 14:58:10.667252 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.667237 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:58:10.673139 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.673102 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:58:10.921288 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.921195 2574 generic.go:356] "Generic (PLEG): container finished" podID="ddab9016-d633-455c-a5e4-477bad67508d" containerID="72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe" exitCode=0 Nov 02 14:58:10.921688 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.921292 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerDied","Data":"72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe"} Nov 02 14:58:10.921688 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:10.921590 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerStarted","Data":"b7d549e975284753735ba54b893e6a085455f508ae6ee948a321e2b5c09e20da"} Nov 02 14:58:11.048821 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:11.048750 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f9zqb"] Nov 02 14:58:11.051267 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:58:11.051243 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13823b41_3da3_4a0b_bf1d_4c9d1b24608f.slice/crio-02faab5193a916b5ef7c5466d24a2707bd9404c9b1a93120312555fe5fcdfc39 WatchSource:0}: Error finding container 02faab5193a916b5ef7c5466d24a2707bd9404c9b1a93120312555fe5fcdfc39: Status 404 returned error can't find the container with id 02faab5193a916b5ef7c5466d24a2707bd9404c9b1a93120312555fe5fcdfc39 Nov 02 14:58:11.068201 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:11.068176 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vjfjv"] Nov 02 14:58:11.094909 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:58:11.094882 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfe09e4_8fce_4149_a9fb_0a222124f121.slice/crio-800a8a1c60f253271b08cf2ada505ddc4ba0a75b7b9b493af95cbff3bfb79474 WatchSource:0}: Error finding container 800a8a1c60f253271b08cf2ada505ddc4ba0a75b7b9b493af95cbff3bfb79474: Status 404 returned error can't find the container with id 800a8a1c60f253271b08cf2ada505ddc4ba0a75b7b9b493af95cbff3bfb79474 Nov 02 14:58:11.104775 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:11.104752 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vxjb7"] Nov 02 14:58:11.110972 ip-10-0-142-141 kubenswrapper[2574]: W1102 14:58:11.110951 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3427e5f_57ef_48d9_ac0f_2b573fb2e8d9.slice/crio-232b81f9ec6c96276ae0a372ac5f67416c0004b526d4db1d0fbcbd1b9b4a5506 WatchSource:0}: Error finding container 232b81f9ec6c96276ae0a372ac5f67416c0004b526d4db1d0fbcbd1b9b4a5506: Status 404 returned error can't find the container with id 232b81f9ec6c96276ae0a372ac5f67416c0004b526d4db1d0fbcbd1b9b4a5506 Nov 02 14:58:11.925918 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:11.925874 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vjfjv" event={"ID":"4bfe09e4-8fce-4149-a9fb-0a222124f121","Type":"ContainerStarted","Data":"800a8a1c60f253271b08cf2ada505ddc4ba0a75b7b9b493af95cbff3bfb79474"} Nov 02 14:58:11.927308 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:11.927275 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f9zqb" event={"ID":"13823b41-3da3-4a0b-bf1d-4c9d1b24608f","Type":"ContainerStarted","Data":"02faab5193a916b5ef7c5466d24a2707bd9404c9b1a93120312555fe5fcdfc39"} Nov 02 14:58:11.928527 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:11.928505 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vxjb7" event={"ID":"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9","Type":"ContainerStarted","Data":"232b81f9ec6c96276ae0a372ac5f67416c0004b526d4db1d0fbcbd1b9b4a5506"} Nov 02 14:58:13.940587 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:13.940545 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f9zqb" event={"ID":"13823b41-3da3-4a0b-bf1d-4c9d1b24608f","Type":"ContainerStarted","Data":"7f9f762e467e23b215b41d4a10672b803de8aad38f5a29cf32a621cacce389ed"} Nov 02 14:58:13.941106 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:13.940616 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f9zqb" event={"ID":"13823b41-3da3-4a0b-bf1d-4c9d1b24608f","Type":"ContainerStarted","Data":"f82c2e124cdcba35bd1fda36c5cb5732dd28807d9013627395ed364326b17bc0"} Nov 02 14:58:15.957095 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:15.956914 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vxjb7" event={"ID":"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9","Type":"ContainerStarted","Data":"872439c1beb658c0b5c9b495d13561ef35447655efd20c80c98a23ddf19c68d5"} Nov 02 14:58:15.958103 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:15.958029 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:58:15.962029 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:15.961985 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerStarted","Data":"fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182"} Nov 02 14:58:15.962159 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:15.962073 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerStarted","Data":"ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b"} Nov 02 14:58:16.158472 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:16.158350 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f9zqb" podStartSLOduration=69.154045715 podStartE2EDuration="1m11.158328964s" totalImagesPullingTime="2.004283249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:14.067552365 +0000 UTC m=+70.032534879" watchObservedRunningTime="2026-11-02 14:58:16.158328964 +0000 UTC m=+72.123311476" Nov 02 14:58:16.158922 ip-10-0-142-141 kubenswrapper[2574]: I1102 14:58:16.158877 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vxjb7" podStartSLOduration=66.532963879 podStartE2EDuration="1m11.158868356s" totalImagesPullingTime="4.625904477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:16.151062676 +0000 UTC m=+72.116045192" watchObservedRunningTime="2026-11-02 14:58:16.158868356 +0000 UTC m=+72.123850872" Nov 02 14:58:17.731716 ip-10-0-142-141 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 14:58:17.752430 ip-10-0-142-141 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 14:58:17.752620 ip-10-0-142-141 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 14:58:17.752699 ip-10-0-142-141 systemd[1]: kubelet.service: Consumed 3.980s CPU time. Nov 02 14:58:17.764705 ip-10-0-142-141 systemd[1]: Starting Kubernetes Kubelet... Nov 02 14:58:17.830213 ip-10-0-142-141 kubenswrapper[5476]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:58:17.830213 ip-10-0-142-141 kubenswrapper[5476]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 14:58:17.830213 ip-10-0-142-141 kubenswrapper[5476]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:58:17.830213 ip-10-0-142-141 kubenswrapper[5476]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:58:17.832258 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832168 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:17.832258 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832252 5476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:17.832258 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832256 5476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:17.832258 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832259 5476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:17.832258 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832263 5476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:17.832258 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832266 5476 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832269 5476 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832272 5476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832274 5476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832277 5476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832280 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832283 5476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832285 5476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832288 5476 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832291 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832293 5476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832296 5476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832299 5476 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832301 5476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832303 5476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832306 5476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832308 5476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832311 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832314 5476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832316 5476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:17.832487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832319 5476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832327 5476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832330 5476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832332 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832337 5476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832341 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832345 5476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832348 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832351 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832354 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832356 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832359 5476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832362 5476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832365 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832367 5476 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832370 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832372 5476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832375 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832378 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:17.832975 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832381 5476 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832384 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832386 5476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832389 5476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832391 5476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832394 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832397 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832399 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832402 5476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832405 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832409 5476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832412 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832416 5476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832420 5476 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832423 5476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832425 5476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832428 5476 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832431 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832437 5476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:17.833478 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832439 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832442 5476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832445 5476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832447 5476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832450 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832453 5476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832455 5476 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832458 5476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832461 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832463 5476 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832466 5476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832468 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832471 5476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832474 5476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832477 5476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832480 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832482 5476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832485 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832487 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:17.833958 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832490 5476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832493 5476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832496 5476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832498 5476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832501 5476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832504 5476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832506 5476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832509 5476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832511 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832514 5476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832516 5476 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832520 5476 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832524 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832526 5476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832529 5476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832532 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832534 5476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832537 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832539 5476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832542 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:17.834487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832544 5476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832547 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832551 5476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.832553 5476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833323 5476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833331 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833334 5476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833337 5476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833340 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833343 5476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833345 5476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833348 5476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833350 5476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833353 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833356 5476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833358 5476 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833362 5476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833364 5476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833367 5476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:17.834987 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833369 5476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833372 5476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833375 5476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833379 5476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833383 5476 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833389 5476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833392 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833395 5476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833398 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833401 5476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833403 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833406 5476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833409 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833411 5476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833414 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833417 5476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833419 5476 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833422 5476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833425 5476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:17.835525 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833427 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833430 5476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833433 5476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833436 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833439 5476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833441 5476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833444 5476 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833446 5476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833449 5476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833451 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833454 5476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833456 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833459 5476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833462 5476 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833464 5476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833467 5476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833470 5476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833473 5476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833478 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:17.835988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833481 5476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833484 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833487 5476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833490 5476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833492 5476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833495 5476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833498 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833500 5476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833503 5476 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833505 5476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833508 5476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833510 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833513 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833516 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833518 5476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833521 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833524 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833526 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833530 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833532 5476 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:17.836487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833536 5476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833540 5476 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833543 5476 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833545 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833548 5476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833551 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833553 5476 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833556 5476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833558 5476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833562 5476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833565 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833569 5476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833571 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833574 5476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833577 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833579 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833582 5476 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833584 5476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833587 5476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833590 5476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:17.837034 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833592 5476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833595 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833597 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833600 5476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833603 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833605 5476 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833608 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833610 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833613 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833616 5476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833619 5476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833621 5476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.833624 5476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834134 5476 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834141 5476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834145 5476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834148 5476 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834151 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834153 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834156 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:17.837578 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834159 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834162 5476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834166 5476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834171 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834174 5476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834177 5476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834179 5476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834182 5476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834185 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834188 5476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834190 5476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834193 5476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834195 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834198 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834201 5476 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834205 5476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834208 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834212 5476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834216 5476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:17.838101 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834219 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834221 5476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834225 5476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834228 5476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834231 5476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834233 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834236 5476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834238 5476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834241 5476 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834244 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834246 5476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834249 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834251 5476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834254 5476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834257 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834259 5476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834263 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834266 5476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834269 5476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:17.838566 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834271 5476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834274 5476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834277 5476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834280 5476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834282 5476 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834285 5476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834288 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834291 5476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834293 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834296 5476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834299 5476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834301 5476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834304 5476 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834306 5476 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834309 5476 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834311 5476 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834315 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834317 5476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834320 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834323 5476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:17.839046 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834325 5476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834328 5476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834331 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834333 5476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834336 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834338 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834341 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834343 5476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834346 5476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834350 5476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834353 5476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834356 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834358 5476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834361 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834363 5476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834366 5476 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834369 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834371 5476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834374 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834376 5476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:17.839545 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834379 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834381 5476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834384 5476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834386 5476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834389 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834391 5476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834394 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834397 5476 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834399 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834402 5476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834405 5476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834408 5476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834410 5476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834413 5476 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834415 5476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834418 5476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834420 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834423 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834426 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:17.840061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834428 5476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.834433 5476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834526 5476 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834539 5476 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834549 5476 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834558 5476 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834567 5476 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834571 5476 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834575 5476 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834580 5476 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834583 5476 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834586 5476 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834589 5476 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834593 5476 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834596 5476 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834599 5476 flags.go:64] FLAG: --cgroup-root="" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834602 5476 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834605 5476 flags.go:64] FLAG: --client-ca-file="" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834608 5476 flags.go:64] FLAG: --cloud-provider="external" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834611 5476 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834615 5476 flags.go:64] FLAG: --cluster-domain="" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834618 5476 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834621 5476 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 14:58:17.840673 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834626 5476 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834631 5476 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834636 5476 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834639 5476 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834642 5476 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834645 5476 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834649 5476 flags.go:64] FLAG: --contention-profiling="false" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834652 5476 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834655 5476 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834658 5476 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834661 5476 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834667 5476 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834670 5476 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834673 5476 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834676 5476 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834679 5476 flags.go:64] FLAG: --enable-server="true" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834682 5476 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834686 5476 flags.go:64] FLAG: --event-burst="100" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834690 5476 flags.go:64] FLAG: --event-qps="50" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834692 5476 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834696 5476 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834699 5476 flags.go:64] FLAG: --eviction-hard="" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834703 5476 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834705 5476 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834709 5476 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 14:58:17.841273 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834711 5476 flags.go:64] FLAG: --eviction-soft="" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834715 5476 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834718 5476 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834721 5476 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834724 5476 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834727 5476 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834730 5476 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834733 5476 flags.go:64] FLAG: --feature-gates="" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834739 5476 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834742 5476 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834745 5476 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834749 5476 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834752 5476 flags.go:64] FLAG: --healthz-port="10248" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834755 5476 flags.go:64] FLAG: --help="false" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834758 5476 flags.go:64] FLAG: --hostname-override="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834761 5476 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834764 5476 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834767 5476 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834771 5476 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834775 5476 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834778 5476 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834781 5476 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834784 5476 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 14:58:17.841914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834787 5476 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834790 5476 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834793 5476 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834796 5476 flags.go:64] FLAG: --kube-reserved="" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834799 5476 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834802 5476 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834805 5476 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834809 5476 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834812 5476 flags.go:64] FLAG: --lock-file="" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834814 5476 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834818 5476 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834821 5476 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834826 5476 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834829 5476 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834832 5476 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834835 5476 flags.go:64] FLAG: --logging-format="text" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834838 5476 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834843 5476 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834847 5476 flags.go:64] FLAG: --manifest-url="" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834850 5476 flags.go:64] FLAG: --manifest-url-header="" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834855 5476 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834858 5476 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834862 5476 flags.go:64] FLAG: --max-pods="110" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834865 5476 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834868 5476 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 14:58:17.842686 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834871 5476 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834874 5476 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834878 5476 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834881 5476 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834884 5476 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834890 5476 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834893 5476 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834896 5476 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834900 5476 flags.go:64] FLAG: --pod-cidr="" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834903 5476 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834906 5476 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834909 5476 flags.go:64] FLAG: --pods-per-core="0" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834912 5476 flags.go:64] FLAG: --port="10250" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834916 5476 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834919 5476 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0cf104e5ef24ae7f0" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834922 5476 flags.go:64] FLAG: --qos-reserved="" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834925 5476 flags.go:64] FLAG: --read-only-port="10255" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834928 5476 flags.go:64] FLAG: --register-node="true" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834931 5476 flags.go:64] FLAG: --register-with-taints="" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834934 5476 flags.go:64] FLAG: --registry-burst="10" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834937 5476 flags.go:64] FLAG: --registry-qps="5" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834940 5476 flags.go:64] FLAG: --reserved-cpus="" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834944 5476 flags.go:64] FLAG: --reserved-memory="" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834947 5476 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834952 5476 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834955 5476 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 14:58:17.843662 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834959 5476 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834962 5476 flags.go:64] FLAG: --runonce="false" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834965 5476 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834968 5476 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834971 5476 flags.go:64] FLAG: --seccomp-default="false" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834974 5476 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834977 5476 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834980 5476 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834983 5476 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.834987 5476 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835002 5476 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835006 5476 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835009 5476 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835012 5476 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835016 5476 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835018 5476 flags.go:64] FLAG: --system-cgroups="" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835022 5476 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835027 5476 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835030 5476 flags.go:64] FLAG: --tls-cert-file="" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835033 5476 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835037 5476 flags.go:64] FLAG: --tls-min-version="" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835040 5476 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835043 5476 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835046 5476 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835049 5476 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 14:58:17.844716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835052 5476 flags.go:64] FLAG: --v="2" Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835056 5476 flags.go:64] FLAG: --version="false" Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835061 5476 flags.go:64] FLAG: --vmodule="" Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835065 5476 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835069 5476 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835218 5476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835224 5476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835227 5476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835230 5476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835233 5476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835235 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835238 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835241 5476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835243 5476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835246 5476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835249 5476 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835251 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835254 5476 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835257 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835260 5476 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:17.845822 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835262 5476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835265 5476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835267 5476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835270 5476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835273 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835275 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835278 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835280 5476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835283 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835286 5476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835288 5476 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835291 5476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835293 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835296 5476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835298 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835301 5476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835303 5476 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835307 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835312 5476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835314 5476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:17.846661 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835317 5476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835320 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835323 5476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835326 5476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835328 5476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835331 5476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835334 5476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835336 5476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835339 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835341 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835344 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835346 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835350 5476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835353 5476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835357 5476 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835360 5476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835362 5476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835365 5476 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835368 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:17.847261 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835370 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835373 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835375 5476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835378 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835380 5476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835383 5476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835386 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835389 5476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835391 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835394 5476 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835398 5476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835402 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835405 5476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835408 5476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835410 5476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835414 5476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835416 5476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835419 5476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835421 5476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:17.847734 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835424 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835426 5476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835429 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835432 5476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835434 5476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835437 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835439 5476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835442 5476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835444 5476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835447 5476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835450 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835452 5476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835455 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835458 5476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835460 5476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835463 5476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835465 5476 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835468 5476 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835471 5476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835473 5476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:17.848210 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835476 5476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835479 5476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835482 5476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835487 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835491 5476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835494 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835497 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835499 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835502 5476 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835505 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835508 5476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835510 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835513 5476 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.835541 5476 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835583 5476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835587 5476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:17.848717 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835591 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835593 5476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835596 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835599 5476 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835601 5476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835604 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835607 5476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835610 5476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835612 5476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835615 5476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835618 5476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835621 5476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835623 5476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835626 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835628 5476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835631 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835633 5476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835636 5476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835638 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:17.849138 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835642 5476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835645 5476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835649 5476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835652 5476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835654 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835657 5476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835660 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835663 5476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835665 5476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835668 5476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835671 5476 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835673 5476 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835676 5476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835678 5476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835681 5476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835683 5476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835688 5476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835692 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835695 5476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:17.849757 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835698 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835701 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835703 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835706 5476 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835708 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835711 5476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835714 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835716 5476 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835719 5476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835721 5476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835724 5476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835727 5476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835729 5476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835733 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835737 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835742 5476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835745 5476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835747 5476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835750 5476 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835753 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:17.850645 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835755 5476 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835758 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835761 5476 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835764 5476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835766 5476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835769 5476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835771 5476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835774 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835777 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835780 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835782 5476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835784 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835787 5476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835790 5476 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835792 5476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835794 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835797 5476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835800 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835802 5476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835805 5476 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:17.851175 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835808 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835810 5476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835813 5476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835815 5476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835818 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835822 5476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835824 5476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835828 5476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835831 5476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835833 5476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835836 5476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835838 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835841 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835844 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835846 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835851 5476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835878 5476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835883 5476 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835886 5476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:17.851666 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835889 5476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835891 5476 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835894 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835896 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835899 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835903 5476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.835905 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.839567 5476 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.839581 5476 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.839600 5476 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.839605 5476 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839668 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839675 5476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839678 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839681 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839684 5476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839687 5476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839690 5476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839693 5476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:17.852145 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839695 5476 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839698 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839701 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839704 5476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839706 5476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839709 5476 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839713 5476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839716 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839719 5476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839722 5476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839725 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839728 5476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839730 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839733 5476 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839736 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839738 5476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839741 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839744 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839747 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:17.852615 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839750 5476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839753 5476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839757 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839761 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839764 5476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839767 5476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839771 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839773 5476 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839776 5476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839779 5476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839782 5476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839785 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839787 5476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839790 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839793 5476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839796 5476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839798 5476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839801 5476 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839804 5476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:17.853129 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839807 5476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839810 5476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839813 5476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839816 5476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839818 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839821 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839824 5476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839826 5476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839829 5476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839831 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839833 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839836 5476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839839 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839842 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839845 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839847 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839850 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839852 5476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839855 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:17.853575 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839858 5476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839860 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839863 5476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839865 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839868 5476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839871 5476 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839873 5476 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839876 5476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839878 5476 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839881 5476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839885 5476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839888 5476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839892 5476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839895 5476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839898 5476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839901 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839905 5476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839908 5476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839911 5476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839913 5476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:17.854061 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839916 5476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839918 5476 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839921 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839923 5476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839926 5476 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839929 5476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839932 5476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839934 5476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839937 5476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839939 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839942 5476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839945 5476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839948 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839950 5476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839953 5476 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839955 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839958 5476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839961 5476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839963 5476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839966 5476 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:17.854559 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.839968 5476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.840015 5476 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840159 5476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840164 5476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840167 5476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840171 5476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840174 5476 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840177 5476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840179 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840182 5476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840185 5476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840187 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840191 5476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840194 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840196 5476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840199 5476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:17.855099 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840202 5476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840204 5476 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840207 5476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840209 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840212 5476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840214 5476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840217 5476 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840219 5476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840222 5476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840225 5476 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840228 5476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840230 5476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840233 5476 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840235 5476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840238 5476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840240 5476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840243 5476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840245 5476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840248 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840250 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:17.855499 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840253 5476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840256 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840259 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840262 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840264 5476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840267 5476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840269 5476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840272 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840275 5476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840278 5476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840280 5476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840283 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840286 5476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840288 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840292 5476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840295 5476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840298 5476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840301 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840304 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:17.856088 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840306 5476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840309 5476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840312 5476 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840314 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840317 5476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840319 5476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840322 5476 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840324 5476 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840327 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840329 5476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840332 5476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840334 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840337 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840339 5476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840342 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840344 5476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840347 5476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840349 5476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840352 5476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840354 5476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:17.856695 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840357 5476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840360 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840364 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840366 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840369 5476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840372 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840374 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840377 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840379 5476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840382 5476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840384 5476 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840387 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840389 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840392 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840394 5476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840397 5476 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840399 5476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840402 5476 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840404 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840407 5476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:17.857303 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840409 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840412 5476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840414 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840417 5476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840419 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840422 5476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840425 5476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840427 5476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840430 5476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840433 5476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840437 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840440 5476 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840443 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.840466 5476 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840498 5476 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840502 5476 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:58:17.857951 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840505 5476 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840508 5476 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840511 5476 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840513 5476 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840516 5476 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840518 5476 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840521 5476 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840524 5476 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840526 5476 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840529 5476 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840531 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840534 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840536 5476 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840538 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840541 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840544 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840546 5476 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840549 5476 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840551 5476 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:58:17.858487 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840554 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840556 5476 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840559 5476 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840561 5476 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840564 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840567 5476 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840569 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840573 5476 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840577 5476 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840580 5476 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840584 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840589 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840593 5476 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840595 5476 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840598 5476 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840601 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840604 5476 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840606 5476 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840609 5476 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840611 5476 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:58:17.859094 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840614 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840616 5476 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840619 5476 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840621 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840624 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840626 5476 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840629 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840632 5476 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840634 5476 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840637 5476 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840639 5476 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840642 5476 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840644 5476 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840647 5476 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840651 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840653 5476 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840656 5476 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840658 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840661 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:58:17.859771 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840663 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840666 5476 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840668 5476 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840671 5476 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840675 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840677 5476 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840681 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840683 5476 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840686 5476 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840689 5476 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840691 5476 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840694 5476 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840696 5476 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840699 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840702 5476 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840705 5476 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840708 5476 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840710 5476 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840713 5476 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840715 5476 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:58:17.860381 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840718 5476 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840721 5476 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840724 5476 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840728 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840731 5476 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840734 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840737 5476 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840740 5476 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840743 5476 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840745 5476 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840748 5476 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840751 5476 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840753 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840755 5476 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840758 5476 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840760 5476 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840764 5476 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840767 5476 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840769 5476 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:58:17.860988 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840773 5476 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840776 5476 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840779 5476 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840782 5476 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840784 5476 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840787 5476 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:17.840790 5476 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.840839 5476 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.841792 5476 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.841846 5476 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.842042 5476 server.go:1014] "Starting client certificate rotation" Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.842177 5476 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.842348 5476 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:52:04 +0000 UTC" deadline="2028-04-23 23:13:59.0994818 +0000 UTC" Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.842376 5476 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12920h15m41.257108257s" Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.844204 5476 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.845979 5476 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:58:17.861746 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.847519 5476 log.go:25] "Validated CRI v1 runtime API" Nov 02 14:58:17.862509 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.849404 5476 log.go:25] "Validated CRI v1 image API" Nov 02 14:58:17.862509 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.850209 5476 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 14:58:17.862509 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.852375 5476 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ec76a8f-cadb-4acf-9be3-29e302aa1b3e:/dev/nvme0n1p3 fcd1a2af-8fc8-4a67-bb90-768d514bcbd0:/dev/nvme0n1p4] Nov 02 14:58:17.862632 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.852396 5476 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02faab5193a916b5ef7c5466d24a2707bd9404c9b1a93120312555fe5fcdfc39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02faab5193a916b5ef7c5466d24a2707bd9404c9b1a93120312555fe5fcdfc39/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/057b1b17adaf5c318b46dee2951fccd0646743349eb73780ec96eed240d7eba9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/057b1b17adaf5c318b46dee2951fccd0646743349eb73780ec96eed240d7eba9/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/232b81f9ec6c96276ae0a372ac5f67416c0004b526d4db1d0fbcbd1b9b4a5506/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/232b81f9ec6c96276ae0a372ac5f67416c0004b526d4db1d0fbcbd1b9b4a5506/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25f28cedced0149eb0f8df7da6abed2fea8378201dc05ef9f35438b58942de77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25f28cedced0149eb0f8df7da6abed2fea8378201dc05ef9f35438b58942de77/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36a5aeef46f3c7ff8afbb40a561ff833653c954f6763644ecedb4d2cb9044704/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36a5aeef46f3c7ff8afbb40a561ff833653c954f6763644ecedb4d2cb9044704/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4611b7d9a9374bac69debb4ab4817106ff8e10f9c8514bad2b93228be942d3e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4611b7d9a9374bac69debb4ab4817106ff8e10f9c8514bad2b93228be942d3e6/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5015acc47cf4b5020f7b0f0e4faf6d8dc1c63d0b7daa01fa7708869ebc020ac6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5015acc47cf4b5020f7b0f0e4faf6d8dc1c63d0b7daa01fa7708869ebc020ac6/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5aedfa6ce17b0d33e1957bf4174f6e175ce936cab0557a8516b5e02adeec7d71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5aedfa6ce17b0d33e1957bf4174f6e175ce936cab0557a8516b5e02adeec7d71/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ee9fd38117b061d44bdb2dfc5cc601cbc938be4d997d0f5c323407dd3d8dd2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ee9fd38117b061d44bdb2dfc5cc601cbc938be4d997d0f5c323407dd3d8dd2f/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/800a8a1c60f253271b08cf2ada505ddc4ba0a75b7b9b493af95cbff3bfb79474/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/800a8a1c60f253271b08cf2ada505ddc4ba0a75b7b9b493af95cbff3bfb79474/userdata/shm major:0 minor:374 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8aa5f84021e0925b39030f1e4c7ff4a7eb3179705920bc780302f2915aa6205d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8aa5f84021e0925b39030f1e4c7ff4a7eb3179705920bc780302f2915aa6205d/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9932e1dc75fd99bb713c16465e83cf2f113d3fcfea5ae912e13fa8c42434a2e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9932e1dc75fd99bb713c16465e83cf2f113d3fcfea5ae912e13fa8c42434a2e9/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a75e8b0581b792e612a420f7c4bfdcb2db686b6af2d26b1ed651279f6dd9461a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a75e8b0581b792e612a420f7c4bfdcb2db686b6af2d26b1ed651279f6dd9461a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae112bf9afbf161f0542ba34faad93de100d37b0ff254d7d17bacc6286ce8be0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae112bf9afbf161f0542ba34faad93de100d37b0ff254d7d17bacc6286ce8be0/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0e7b5b1f713c379c6ad72e211f011a4861cea6c16607f5c8be4f71aecfbb0d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0e7b5b1f713c379c6ad72e211f011a4861cea6c16607f5c8be4f71aecfbb0d6/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7d549e975284753735ba54b893e6a085455f508ae6ee948a321e2b5c09e20da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7d549e975284753735ba54b893e6a085455f508ae6ee948a321e2b5c09e20da/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8371c5f9d1ab400e9147eec249f811122ae10c36857b0842a7a09ed2870e9ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8371c5f9d1ab400e9147eec249f811122ae10c36857b0842a7a09ed2870e9ce/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c33a739d8882496964cbd135a96dbe0c794add472b1fa44e7ae8d4809d0dd321/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c33a739d8882496964cbd135a96dbe0c794add472b1fa44e7ae8d4809d0dd321/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf603277525bbb9571d2fdcc94ee1ec1f423ca8841deea710e422425860fbd44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf603277525bbb9571d2fdcc94ee1ec1f423ca8841deea710e422425860fbd44/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d38fb6ced2844cadde02cd689ad85b4ce843ca0b911e766acfa0be64aed8c918/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d38fb6ced2844cadde02cd689ad85b4ce843ca0b911e766acfa0be64aed8c918/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6a70d6e7a0d42f7702cdb67e0bdfe3cbf436f17b1c762ac22bbd768071cc2ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6a70d6e7a0d42f7702cdb67e0bdfe3cbf436f17b1c762ac22bbd768071cc2ce/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0444057f-0080-4135-9d83-dca94858033e/volumes/kubernetes.io~projected/kube-api-access-wbr8c:{mountpoint:/var/lib/kubelet/pods/0444057f-0080-4135-9d83-dca94858033e/volumes/kubernetes.io~projected/kube-api-access-wbr8c major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13823b41-3da3-4a0b-bf1d-4c9d1b24608f/volumes/kubernetes.io~projected/kube-api-access-29cc4:{mountpoint:/var/lib/kubelet/pods/13823b41-3da3-4a0b-bf1d-4c9d1b24608f/volumes/kubernetes.io~projected/kube-api-access-29cc4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13823b41-3da3-4a0b-bf1d-4c9d1b24608f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/13823b41-3da3-4a0b-bf1d-4c9d1b24608f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25f92d33-6475-49f1-bfff-691a5dec0ae0/volumes/kubernetes.io~projected/kube-api-access-wcklx:{mountpoint:/var/lib/kubelet/pods/25f92d33-6475-49f1-bfff-691a5dec0ae0/volumes/kubernetes.io~projected/kube-api-access-wcklx major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25f92d33-6475-49f1-bfff-691a5dec0ae0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/25f92d33-6475-49f1-bfff-691a5dec0ae0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e7e4e7b-c190-420d-b438-b2c9e1074795/volumes/kubernetes.io~projected/kube-api-access-j698s:{mountpoint:/var/lib/kubelet/pods/2e7e4e7b-c190-420d-b438-b2c9e1074795/volumes/kubernetes.io~projected/kube-api-access-j698s major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/449bb762-da7a-4a88-9796-1a1ef23e7c8c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/449bb762-da7a-4a88-9796-1a1ef23e7c8c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/449bb762-da7a-4a88-9796-1a1ef23e7c8c/volumes/kubernetes.io~projected/kube-api-access-x5hh7:{mountpoint:/var/lib/kubelet/pods/449bb762-da7a-4a88-9796-1a1ef23e7c8c/volumes/kubernetes.io~projected/kube-api-access-x5hh7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/449bb762-da7a-4a88-9796-1a1ef23e7c8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/449bb762-da7a-4a88-9796-1a1ef23e7c8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bfe09e4-8fce-4149-a9fb-0a222124f121/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4bfe09e4-8fce-4149-a9fb-0a222124f121/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ffbd00b-29ae-4d63-97b8-edf990224dcd/volumes/kubernetes.io~projected/kube-api-access-7hzs7:{mountpoint:/var/lib/kubelet/pods/4ffbd00b-29ae-4d63-97b8-edf990224dcd/volumes/kubernetes.io~projected/kube-api-access-7hzs7 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~projected/kube-api-access-fmc4m:{mountpoint:/var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~projected/kube-api-access-fmc4m major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66740b77-b82e-49c2-b8d0-af48d25147cb/volumes/kubernetes.io~projected/kube-api-access-2slcd:{mountpoint:/var/lib/kubelet/pods/66740b77-b82e-49c2-b8d0-af48d25147cb/volumes/kubernetes.io~projected/kube-api-access-2slcd major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66740b77-b82e-49c2-b8d0-af48d25147cb/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/66740b77-b82e-49c2-b8d0-af48d25147cb/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8880c66c-32b2-4a81-a174-34242093784d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8880c66c-32b2-4a81-a174-34242093784d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8880c66c-32b2-4a81-a174-34242093784d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8880c66c-32b2-4a81-a174-34242093784d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8880c66c-32b2-4a81-a174-34242093784d/volumes/kubernetes.io~projected/kube-api-access-r4p4n:{mountpoint:/var/lib/kubelet/pods/8880c66c-32b2-4a81-a174-34242093784d/volumes/kubernetes.io~projected/kube-api-access-r4p4n major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/989be356-90d5-4c62-8200-5d5a8d2af857/volumes/kubernetes.io~projected/kube-api-access-jsfph:{mountpoint:/var/lib/kubelet/pods/989be356-90d5-4c62-8200-5d5a8d2af857/volumes/kubernetes.io~projected/kube-api-access-jsfph major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/989be356-90d5-4c62-8200-5d5a8d2af857/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/989be356-90d5-4c62-8200-5d5a8d2af857/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3d8b64c-559c-4806-8599-25b62c630fb4/volumes/kubernetes.io~projected/kube-api-access-lnvct:{mountpoint:/var/lib/kubelet/pods/b3d8b64c-559c-4806-8599-25b62c630fb4/volumes/kubernetes.io~projected/kube-api-access-lnvct major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5cc5088-ab54-47e3-bcca-7cb792f88880/volumes/kubernetes.io~projected/kube-api-access-lr7rp:{mountpoint:/var/lib/kubelet/pods/b5cc5088-ab54-47e3-bcca-7cb792f88880/volumes/kubernetes.io~projected/kube-api-access-lr7rp major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/web-config/alertmanager/9 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/web-config/config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~empty-dir/config-out major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~projected/kube-api-access-8zjwc:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~projected/kube-api-access-8zjwc major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~projected/tls-assets major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/config-volume major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/web-config major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d186e7b3-d2fa-4f5b-bd31-ee633608c8db/volumes/kubernetes.io~projected/kube-api-access-hz25j:{mountpoint:/var/lib/kubelet/pods/d186e7b3-d2fa-4f5b-bd31-ee633608c8db/volumes/kubernetes.io~projected/kube-api-access-hz25j major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d186e7b3-d2fa-4f5b-bd31-ee633608c8db/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d186e7b3-d2fa-4f5b-bd31-ee633608c8db/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d186e7b3-d2fa-4f5b-bd31-ee633608c8db/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d186e7b3-d2fa-4f5b-bd31-ee633608c8db/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1af9240-8442-4911-b1d9-5570aea28abf/volumes/kubernetes.io~projected/kube-api-access-smgsz:{mountpoint:/var/lib/kubelet/pods/d1af9240-8442-4911-b1d9-5570aea28abf/volumes/kubernetes.io~projected/kube-api-access-smgsz major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1af9240-8442-4911-b1d9-5570aea28abf/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d1af9240-8442-4911-b1d9-5570aea28abf/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1af9240-8442-4911-b1d9-5570aea28abf/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/d1af9240-8442-4911-b1d9-5570aea28abf/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9/volumes/kubernetes.io~projected/kube-api-access-6mjmh:{mountpoint:/var/lib/kubelet/pods/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9/volumes/kubernetes.io~projected/kube-api-access-6mjmh major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volume-subpaths/web-config/config-reloader/2 major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volume-subpaths/web-config/prometheus/15 major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~projected/kube-api-access-vsxp4:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~projected/kube-api-access-vsxp4 major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~projected/tls-assets major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/config major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/web-config major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efc5a077-d018-4792-8551-b9c3594967dd/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/efc5a077-d018-4792-8551-b9c3594967dd/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752/volumes/kubernetes.io~projected/kube-api-access-p65ch:{mountpoint:/var/lib/kubelet/pods/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752/volumes/kubernetes.io~projected/kube-api-access-p65ch major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/8f22fe841ad72b452a57fe1b9b2b835be44813699a8688a7e2aca5daa7830132/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a7151fce0579d1361aea2d3bcb778a45342bc3d0d55914819ea4a0685988d561/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/29fa6d6a78ac31f6344b411f7094bbf5c81defa53cc63fe7960fdbf81c8b6fad/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/f28296fad677e57231038eb6f715ccc872cb21e3ff840f288e26d99937391b8b/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5eabb5ca4eeb8d3d4da8647ca115e812d5f0ac44af05124b5eb01f6a180dc3ce/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/85a374c83c4f6eedf0323989b4483509973ab554d08794c8f78e9312378f1352/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6232c1691d861ff4141705fd30a759f80257e177eb7f376ec34e4de9b2df11b5/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/d75326ea59954a4c3a6b24659b347e24e1b6cd71b36b5c871f634c2131e771e8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/9cd4cbe908f04b194d7e3a415f2d0b58a818c4b6dad61dc73d1e01a4635f117b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/bb74a64095668f80ddffd91079637d74ac0d34e99f094f6f2c4b2faafdc11739/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/527c2c6d5589efa4bbb7690fab25c9712f7cd080db7ce9533957b1ea7db3d7ba/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/acce851038263c0ba957a2c0c287762235e9473f27f593df3a8894c9e3d71ae0/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f6890cce0e7b00a0c9fe90f88f3c0a7398e4dd7db646032d504ed06c21efea84/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f0a96da7cab9043c75530b1e4c54195bd2c140b41ae6715ed7b6eb23a9052ddc/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5440d388f508865e667cfd64c56525bc31642dcd753083057e8721201009b691/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/86b52557c7068f4570766e1ea68bd9b3fc9bac26ed804ec0a88e854dff24b1f7/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8d740b45395862bc351f566cc6dfcecbca54fd5a5a9ff8faaacfb1cb3bfcdd79/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/93a74776774e975bb9afb19169c312465a65674a3bf6d5d0d92f2458f5dad657/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/39fe0f2689ea6725fd22357eb59fd679f4d1669dda3af4f055b3a98dcd228864/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/58d6b7e49c0d12019c47eb40cc9a76ce4a35b46e015255fa5535dc567fde5dbc/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ecba513c09bd952a7cdb3c3b84655cfd674756dce207f93c7445b42afa5dca71/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b1015700e0909f985ce98dda87a7af49c3ceec6d526e67a918a29b0454624cc0/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/08eb918c2d12f63790b28a00d885a0a5509a8dd10e36b9487f3c37b5f894bd98/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/4b81617fe60f6c280fcb8a0c1c4339932768a6059591a2822074e45de755745b/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/062992a5eb5e1edc220132f8824815c9df08dea71d295d3d1173271a7a61c0b1/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/5da0460afa1defba73573994ba298a2fb732b90f6b1fe070198019a68eb4afa4/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/9e568635170c3a1bdbd1223f83bb06441a48116ea47ec3771a95ff2ebc91a613/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/01e0b8b18508aa6dacbf0d827972f7b498012f659aaa3895efc473b88e30e70e/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/3264dfc5de9f2f4054ab124cb2bb9a753592c5d4220619a314e0f1679a81f128/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/cc4d58272f32d6ef605596edb0077a6f48cf33f99301a8c0d3e69f2bd9cfc17b/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/975859354f83b23477cbc2f81dd756f4c2a698dd47b217f66694ebd7bbfa252d/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/0117282f03177717807b7816a7f3af1b25836cb8c3158a6a0478e546b9d40950/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/c3cc14546951a3ed51233807f67c0ebfe336b267c3442c6fe1c54ddb50396ae8/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/d5f0272ed398cfb511e897ccaf182f1a4a9ed845f1dda2cf704a7b9107a91a94/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/617f4e06abc1a2f87c6c1eb43ef5fc9d2e09e0ca363f621bd0c25dd68536ba0d/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/9312c46535cddedc22086160e85dfdaff961e0eda6d0ce1230aa45db9f296bf7/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/33747c7100d0242c76a1149a7c9d04a2e211c6dfb907f3971dfc12fb4e35f36f/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/13e7f88ef07d8e385f87cb0a1bdb237d95b9ce7ceaca903fbe0cdb7c8ffa3545/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/38356d4f05013a4ec8d44fed3c05d1fa3522be64c5665195595e84f7269d5e67/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/41531a1c138bd6fead176909e577e11153230fc66dc1b3de60a06e7735dea3e3/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/d5acc06104031c921170577d21d355e440634210d27dff97fcbf6fb4f5fd1b36/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/e77736199b13d97fcc4be3e1357d83135e14d4db8772528c7564633fdc81e8ff/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/9a1df257840f60e4ebc0951a6b92361cbd8ffe038cc6d74635c872188224416a/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/cd1f5bf87b13c2dc6e2eb1c3e42c553f86173febf40c40955796a0336d56952b/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/e3db051adc72b7a4c605ac2f6c45ca7ec391dc00c7c4601b4535ae6d46437ccc/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/3e520b2bc1b2a7e8ea9dc561f2ef873759a5e972e58b29c0988df4f6d3506f9a/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/3f6262aee46a0e0fecc3a4bac8383162f2f8a0b117a4a81032da2bdfc4e41055/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/24e8f297495ded581df4fb41d406545880dbddb2e92fa89c3d40a46119f718a7/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/ee49780d4bd117f68572f85ea2f594fad5816accfe456583cab5416bfb381add/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/5f127011bea5ac832af5a2051c47a0b139299a514faeef6c2570401567b53d44/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/4454655d4cf1d0758ab38be70321c0eb0a60fce2a561541774179862edd429c9/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/e41b0e184788c5961f20f827b24c8d889866fd57ccf0c5f53f1c55fb2d0a95c3/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/ded30c4dff33a7f35b0f7cefb7dafc6f3d79e0b66b9ab5794bdb7791febd8382/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/71e5fb74965420cd22cf05681f6a5280f0e1d28e0830f2b65e67d62151e90657/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/296d3a82891f4b12b31d6151490f7a3670d2d5529e22d979b655ec9b39009b8c/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/a0198ac32b04fddc21f4896ac5e0d0563791fdea23bcb209b6f51063dcaa8b1e/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/73d5035e79ea3e3e4e02b897f3863989a9c0b2a278db0c5e4bbadc1624c1f082/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/b38a96bb5e3657ef79376739c5b1a804fc6d6266c6621953be7c1b24ea375822/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/4ace4acdb71a9215a112bf07cba79634c1daea36ade1c2dbdb3588b04df55da4/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/8e03baed1b413a4a50c9efd7a24d61113d6f2ac70f8020f30fa127691b378164/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7e84a86e537a8e11e98ba798c1efdb99bfbe1b79e64088a973a3d8cd07c59a38/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/013989ec135714f8a49d92d903ff63216d2e22cbf9f726b28677c66e31f7d51b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f1edb039df286d6ccf30767d8757b1436645796421caae7989052bf632088ca7/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/97e963751e5bd1fd3105c2df5caa7fe1553f7e5539bc6f8978d7a558311bb552/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/3092fee01433151052307b416dc6be1b45a85777cffce468348218f5d8b4cc25/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e9e4d7a62a52cf774e0dad4086220a8886497e6cf8a355b556f3b23e8e1d3573/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/afcf5c9d18afef946b5e8ecc80a9123150bf5d9aef132107242b26cb1e93fab4/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.859981 5476 manager.go:217] Machine: {Timestamp:2026-11-02 14:58:17.859446148 +0000 UTC m=+0.062324198 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099836 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec237768f56c812721c45ac86f21b54f SystemUUID:ec237768-f56c-8127-21c4-5ac86f21b54f BootID:eb039694-a253-48d0-b78c-910df1e21446 Filesystems:[{Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9932e1dc75fd99bb713c16465e83cf2f113d3fcfea5ae912e13fa8c42434a2e9/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d186e7b3-d2fa-4f5b-bd31-ee633608c8db/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:262 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:357 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13823b41-3da3-4a0b-bf1d-4c9d1b24608f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/4ffbd00b-29ae-4d63-97b8-edf990224dcd/volumes/kubernetes.io~projected/kube-api-access-7hzs7 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~projected/kube-api-access-fmc4m DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8880c66c-32b2-4a81-a174-34242093784d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/449bb762-da7a-4a88-9796-1a1ef23e7c8c/volumes/kubernetes.io~projected/kube-api-access-x5hh7 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1af9240-8442-4911-b1d9-5570aea28abf/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d186e7b3-d2fa-4f5b-bd31-ee633608c8db/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:261 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:281 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:362 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66740b77-b82e-49c2-b8d0-af48d25147cb/volumes/kubernetes.io~projected/kube-api-access-2slcd DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~projected/kube-api-access-vsxp4 DeviceMajor:0 DeviceMinor:366 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bfe09e4-8fce-4149-a9fb-0a222124f121/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:370 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/232b81f9ec6c96276ae0a372ac5f67416c0004b526d4db1d0fbcbd1b9b4a5506/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efc5a077-d018-4792-8551-b9c3594967dd/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d38fb6ced2844cadde02cd689ad85b4ce843ca0b911e766acfa0be64aed8c918/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6a70d6e7a0d42f7702cdb67e0bdfe3cbf436f17b1c762ac22bbd768071cc2ce/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5015acc47cf4b5020f7b0f0e4faf6d8dc1c63d0b7daa01fa7708869ebc020ac6/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:275 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7d549e975284753735ba54b893e6a085455f508ae6ee948a321e2b5c09e20da/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8aa5f84021e0925b39030f1e4c7ff4a7eb3179705920bc780302f2915aa6205d/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/989be356-90d5-4c62-8200-5d5a8d2af857/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5cc5088-ab54-47e3-bcca-7cb792f88880/volumes/kubernetes.io~projected/kube-api-access-lr7rp DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66740b77-b82e-49c2-b8d0-af48d25147cb/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:352 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0444057f-0080-4135-9d83-dca94858033e/volumes/kubernetes.io~projected/kube-api-access-wbr8c DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8371c5f9d1ab400e9147eec249f811122ae10c36857b0842a7a09ed2870e9ce/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25f92d33-6475-49f1-bfff-691a5dec0ae0/volumes/kubernetes.io~projected/kube-api-access-wcklx DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae112bf9afbf161f0542ba34faad93de100d37b0ff254d7d17bacc6286ce8be0/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8880c66c-32b2-4a81-a174-34242093784d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25f92d33-6475-49f1-bfff-691a5dec0ae0/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:276 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:359 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8880c66c-32b2-4a81-a174-34242093784d/volumes/kubernetes.io~projected/kube-api-access-r4p4n DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c33a739d8882496964cbd135a96dbe0c794add472b1fa44e7ae8d4809d0dd321/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/057b1b17adaf5c318b46dee2951fccd0646743349eb73780ec96eed240d7eba9/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25f28cedced0149eb0f8df7da6abed2fea8378201dc05ef9f35438b58942de77/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/989be356-90d5-4c62-8200-5d5a8d2af857/volumes/kubernetes.io~projected/kube-api-access-jsfph DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5aedfa6ce17b0d33e1957bf4174f6e175ce936cab0557a8516b5e02adeec7d71/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d186e7b3-d2fa-4f5b-bd31-ee633608c8db/volumes/kubernetes.io~projected/kube-api-access-hz25j DeviceMajor:0 DeviceMinor:270 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:365 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:363 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e7e4e7b-c190-420d-b438-b2c9e1074795/volumes/kubernetes.io~projected/kube-api-access-j698s DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9/volumes/kubernetes.io~projected/kube-api-access-6mjmh DeviceMajor:0 DeviceMinor:371 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~projected/kube-api-access-8zjwc DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36a5aeef46f3c7ff8afbb40a561ff833653c954f6763644ecedb4d2cb9044704/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ee9fd38117b061d44bdb2dfc5cc601cbc938be4d997d0f5c323407dd3d8dd2f/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/449bb762-da7a-4a88-9796-1a1ef23e7c8c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d1af9240-8442-4911-b1d9-5570aea28abf/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a75e8b0581b792e612a420f7c4bfdcb2db686b6af2d26b1ed651279f6dd9461a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e8592fc-f083-4eb8-b934-517c47f123c4/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13823b41-3da3-4a0b-bf1d-4c9d1b24608f/volumes/kubernetes.io~projected/kube-api-access-29cc4 DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/449bb762-da7a-4a88-9796-1a1ef23e7c8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1af9240-8442-4911-b1d9-5570aea28abf/volumes/kubernetes.io~projected/kube-api-access-smgsz DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0e7b5b1f713c379c6ad72e211f011a4861cea6c16607f5c8be4f71aecfbb0d6/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02faab5193a916b5ef7c5466d24a2707bd9404c9b1a93120312555fe5fcdfc39/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3d8b64c-559c-4806-8599-25b62c630fb4/volumes/kubernetes.io~projected/kube-api-access-lnvct DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:277 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752/volumes/kubernetes.io~projected/kube-api-access-p65ch DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:364 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/800a8a1c60f253271b08cf2ada505ddc4ba0a75b7b9b493af95cbff3bfb79474/userdata/shm DeviceMajor:0 DeviceMinor:374 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4611b7d9a9374bac69debb4ab4817106ff8e10f9c8514bad2b93228be942d3e6/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf603277525bbb9571d2fdcc94ee1ec1f423ca8841deea710e422425860fbd44/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02faab5193a916b MacAddress:7e:69:e6:ab:24:25 Speed:10000 Mtu:8901} {Name:232b81f9ec6c962 MacAddress:c6:26:f2:a4:84:1e Speed:10000 Mtu:8901} {Name:5015acc47cf4b50 MacAddress:ca:a9:43:8b:59:ad Speed:10000 Mtu:8901} {Name:5aedfa6ce17b0d3 MacAddress:b2:3d:5a:ae:ee:f6 Speed:10000 Mtu:8901} {Name:800a8a1c60f2532 MacAddress:de:1a:d1:74:52:bb Speed:10000 Mtu:8901} {Name:9932e1dc75fd99b MacAddress:8a:91:23:ae:b4:e4 Speed:10000 Mtu:8901} {Name:ae112bf9afbf161 MacAddress:6a:ca:e0:99:63:55 Speed:10000 Mtu:8901} {Name:b0e7b5b1f713c37 MacAddress:3a:71:96:d8:02:de Speed:10000 Mtu:8901} {Name:b7d549e97528475 MacAddress:be:88:b8:9b:c4:8b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d2:ec:15:23 Speed:0 Mtu:9001} {Name:br-int MacAddress:9e:62:be:38:aa:85 Speed:0 Mtu:8901} {Name:cf603277525bbb9 MacAddress:7e:b5:7a:82:ec:7f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d2:ec:15:23 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b6:cb:1e:c8:c1:90 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a2:9c:b4:8e:8e:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.860436 5476 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.860486 5476 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.861112 5476 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.861164 5476 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-141.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.861573 5476 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.861585 5476 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.861601 5476 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.861619 5476 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.861758 5476 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.862245 5476 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.862499 5476 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.862538 5476 kubelet.go:493] "Attempting to sync node with API server" Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.862552 5476 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.862565 5476 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.862575 5476 kubelet.go:405] "Adding apiserver pod source" Nov 02 14:58:17.862902 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.862583 5476 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 14:58:17.863514 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.863317 5476 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 14:58:17.863514 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.863333 5476 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:58:17.863899 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.863882 5476 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 14:58:17.876265 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876249 5476 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 14:58:17.876265 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876263 5476 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:58:17.876376 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876366 5476 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 14:58:17.876570 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876559 5476 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 14:58:17.876600 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876589 5476 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 14:58:17.876708 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876697 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 14:58:17.876740 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876718 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 14:58:17.876740 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876727 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 14:58:17.876806 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876741 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 14:58:17.876806 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876752 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 14:58:17.876806 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876759 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 14:58:17.876806 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876771 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 14:58:17.876806 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876778 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 14:58:17.876806 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876785 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 14:58:17.876806 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876790 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 14:58:17.876806 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876802 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 14:58:17.877074 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876812 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 14:58:17.877074 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876836 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 14:58:17.877074 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.876844 5476 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 14:58:17.880284 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.880270 5476 server.go:1263] "Started kubelet" Nov 02 14:58:17.880592 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.880352 5476 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 14:58:17.880932 ip-10-0-142-141 systemd[1]: Started Kubernetes Kubelet. Nov 02 14:58:17.881102 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.880476 5476 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 14:58:17.882122 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.882098 5476 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 14:58:17.882447 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.882423 5476 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 14:58:17.887523 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.885833 5476 server.go:317] "Adding debug handlers to kubelet server" Nov 02 14:58:17.887523 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.886069 5476 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 14:58:17.887523 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.886496 5476 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 14:58:17.887523 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.886536 5476 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:52:06 +0000 UTC" deadline="2028-07-11 14:50:06.517576642 +0000 UTC" Nov 02 14:58:17.887523 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.886567 5476 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14807h51m48.631012383s" Nov 02 14:58:17.887523 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:17.886641 5476 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:58:17.887523 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.886689 5476 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 14:58:17.887523 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.886696 5476 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 14:58:17.887523 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.886808 5476 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 14:58:17.887523 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.887492 5476 factory.go:55] Registering systemd factory Nov 02 14:58:17.887523 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.887513 5476 factory.go:223] Registration of the systemd container factory successfully Nov 02 14:58:17.889086 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.889059 5476 factory.go:153] Registering CRI-O factory Nov 02 14:58:17.889192 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.889089 5476 factory.go:223] Registration of the crio container factory successfully Nov 02 14:58:17.889192 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.889161 5476 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 14:58:17.889192 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.889187 5476 factory.go:103] Registering Raw factory Nov 02 14:58:17.889326 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.889235 5476 manager.go:1199] Started watching for new ooms in manager Nov 02 14:58:17.889720 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.889704 5476 manager.go:319] Starting recovery of all containers Nov 02 14:58:17.891244 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891204 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0444057f-0080-4135-9d83-dca94858033e" volumeName="kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 14:58:17.891397 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891249 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0444057f-0080-4135-9d83-dca94858033e" volumeName="kubernetes.io/projected/0444057f-0080-4135-9d83-dca94858033e-kube-api-access-wbr8c" seLinuxMountContext="" Nov 02 14:58:17.891397 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891265 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 02 14:58:17.891397 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891277 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d186e7b3-d2fa-4f5b-bd31-ee633608c8db" volumeName="kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 14:58:17.891397 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891291 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1af9240-8442-4911-b1d9-5570aea28abf" volumeName="kubernetes.io/configmap/d1af9240-8442-4911-b1d9-5570aea28abf-metrics-client-ca" seLinuxMountContext="" Nov 02 14:58:17.891397 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891304 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efc5a077-d018-4792-8551-b9c3594967dd" volumeName="kubernetes.io/configmap/efc5a077-d018-4792-8551-b9c3594967dd-konnectivity-ca" seLinuxMountContext="" Nov 02 14:58:17.891397 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891320 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25f92d33-6475-49f1-bfff-691a5dec0ae0" volumeName="kubernetes.io/empty-dir/25f92d33-6475-49f1-bfff-691a5dec0ae0-data-volume" seLinuxMountContext="" Nov 02 14:58:17.891397 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891331 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25f92d33-6475-49f1-bfff-691a5dec0ae0" volumeName="kubernetes.io/secret/25f92d33-6475-49f1-bfff-691a5dec0ae0-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 14:58:17.891397 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891347 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66740b77-b82e-49c2-b8d0-af48d25147cb" volumeName="kubernetes.io/projected/66740b77-b82e-49c2-b8d0-af48d25147cb-kube-api-access-2slcd" seLinuxMountContext="" Nov 02 14:58:17.891397 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891361 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d186e7b3-d2fa-4f5b-bd31-ee633608c8db" volumeName="kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 14:58:17.891397 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891381 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1af9240-8442-4911-b1d9-5570aea28abf" volumeName="kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 14:58:17.891397 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891395 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-metrics-client-ca" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891408 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-out" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891421 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d186e7b3-d2fa-4f5b-bd31-ee633608c8db" volumeName="kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-tls" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891458 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891500 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" volumeName="kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891515 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="989be356-90d5-4c62-8200-5d5a8d2af857" volumeName="kubernetes.io/empty-dir/989be356-90d5-4c62-8200-5d5a8d2af857-tmp-dir" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891526 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d186e7b3-d2fa-4f5b-bd31-ee633608c8db" volumeName="kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-metrics-client-ca" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891536 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752" volumeName="kubernetes.io/projected/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-kube-api-access-p65ch" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891547 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25f92d33-6475-49f1-bfff-691a5dec0ae0" volumeName="kubernetes.io/configmap/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891557 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e7e4e7b-c190-420d-b438-b2c9e1074795" volumeName="kubernetes.io/projected/2e7e4e7b-c190-420d-b438-b2c9e1074795-kube-api-access-j698s" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891569 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891581 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9" volumeName="kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891591 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891601 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891621 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efc5a077-d018-4792-8551-b9c3594967dd" volumeName="kubernetes.io/secret/efc5a077-d018-4792-8551-b9c3594967dd-agent-certs" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891631 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e8592fc-f083-4eb8-b934-517c47f123c4" volumeName="kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-client-tls" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891660 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e8592fc-f083-4eb8-b934-517c47f123c4" volumeName="kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-federate-client-tls" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891678 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8880c66c-32b2-4a81-a174-34242093784d" volumeName="kubernetes.io/projected/8880c66c-32b2-4a81-a174-34242093784d-kube-api-access-r4p4n" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891690 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d8b64c-559c-4806-8599-25b62c630fb4" volumeName="kubernetes.io/empty-dir/b3d8b64c-559c-4806-8599-25b62c630fb4-tmp-dir" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891701 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891712 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d186e7b3-d2fa-4f5b-bd31-ee633608c8db" volumeName="kubernetes.io/empty-dir/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-textfile" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891724 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-grpc-tls" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891737 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="989be356-90d5-4c62-8200-5d5a8d2af857" volumeName="kubernetes.io/configmap/989be356-90d5-4c62-8200-5d5a8d2af857-config-volume" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891748 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-cluster-tls-config" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891763 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-web-config" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891774 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d186e7b3-d2fa-4f5b-bd31-ee633608c8db" volumeName="kubernetes.io/projected/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-kube-api-access-hz25j" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891785 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-kube-api-access-vsxp4" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891796 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="989be356-90d5-4c62-8200-5d5a8d2af857" volumeName="kubernetes.io/projected/989be356-90d5-4c62-8200-5d5a8d2af857-kube-api-access-jsfph" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891807 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891820 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-tls-assets" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891834 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752" volumeName="kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-daemon-config" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891846 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25f92d33-6475-49f1-bfff-691a5dec0ae0" volumeName="kubernetes.io/projected/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-api-access-wcklx" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891859 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449bb762-da7a-4a88-9796-1a1ef23e7c8c" volumeName="kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-env-overrides" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891871 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-tls-assets" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891887 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891902 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-metrics-client-ca" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891914 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-config-out" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891925 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-db" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891937 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449bb762-da7a-4a88-9796-1a1ef23e7c8c" volumeName="kubernetes.io/projected/449bb762-da7a-4a88-9796-1a1ef23e7c8c-kube-api-access-x5hh7" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891952 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e8592fc-f083-4eb8-b934-517c47f123c4" volumeName="kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-metrics-client-ca" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891964 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="989be356-90d5-4c62-8200-5d5a8d2af857" volumeName="kubernetes.io/secret/989be356-90d5-4c62-8200-5d5a8d2af857-metrics-tls" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.891978 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5cc5088-ab54-47e3-bcca-7cb792f88880" volumeName="kubernetes.io/projected/b5cc5088-ab54-47e3-bcca-7cb792f88880-kube-api-access-lr7rp" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892009 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-main-db" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892026 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-kube-api-access-8zjwc" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892039 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892062 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892075 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0444057f-0080-4135-9d83-dca94858033e" volumeName="kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-binary-copy" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892089 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449bb762-da7a-4a88-9796-1a1ef23e7c8c" volumeName="kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-script-lib" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892102 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449bb762-da7a-4a88-9796-1a1ef23e7c8c" volumeName="kubernetes.io/secret/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892117 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e8592fc-f083-4eb8-b934-517c47f123c4" volumeName="kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892130 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1af9240-8442-4911-b1d9-5570aea28abf" volumeName="kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-tls" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892143 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892156 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892172 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-web-config" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892186 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bfe09e4-8fce-4149-a9fb-0a222124f121" volumeName="kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892202 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ffbd00b-29ae-4d63-97b8-edf990224dcd" volumeName="kubernetes.io/projected/4ffbd00b-29ae-4d63-97b8-edf990224dcd-kube-api-access-7hzs7" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892217 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e8592fc-f083-4eb8-b934-517c47f123c4" volumeName="kubernetes.io/projected/5e8592fc-f083-4eb8-b934-517c47f123c4-kube-api-access-fmc4m" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892229 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e8592fc-f083-4eb8-b934-517c47f123c4" volumeName="kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892244 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892258 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-config" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892270 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752" volumeName="kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cni-binary-copy" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892281 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="449bb762-da7a-4a88-9796-1a1ef23e7c8c" volumeName="kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-config" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892292 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e8592fc-f083-4eb8-b934-517c47f123c4" volumeName="kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892305 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892315 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-metrics-client-certs" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892330 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13823b41-3da3-4a0b-bf1d-4c9d1b24608f" volumeName="kubernetes.io/projected/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-kube-api-access-29cc4" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892344 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e8592fc-f083-4eb8-b934-517c47f123c4" volumeName="kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-trusted-ca-bundle" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892375 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8880c66c-32b2-4a81-a174-34242093784d" volumeName="kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-tmp" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892390 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1af9240-8442-4911-b1d9-5570aea28abf" volumeName="kubernetes.io/projected/d1af9240-8442-4911-b1d9-5570aea28abf-kube-api-access-smgsz" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892406 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892421 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0444057f-0080-4135-9d83-dca94858033e" volumeName="kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892436 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e7e4e7b-c190-420d-b438-b2c9e1074795" volumeName="kubernetes.io/configmap/2e7e4e7b-c190-420d-b438-b2c9e1074795-serviceca" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892448 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ffbd00b-29ae-4d63-97b8-edf990224dcd" volumeName="kubernetes.io/configmap/4ffbd00b-29ae-4d63-97b8-edf990224dcd-iptables-alerter-script" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892459 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66740b77-b82e-49c2-b8d0-af48d25147cb" volumeName="kubernetes.io/secret/66740b77-b82e-49c2-b8d0-af48d25147cb-cert" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892469 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8880c66c-32b2-4a81-a174-34242093784d" volumeName="kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-etc-tuned" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892481 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3d8b64c-559c-4806-8599-25b62c630fb4" volumeName="kubernetes.io/projected/b3d8b64c-559c-4806-8599-25b62c630fb4-kube-api-access-lnvct" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892491 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" volumeName="kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-volume" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892528 5476 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddab9016-d633-455c-a5e4-477bad67508d" volumeName="kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892542 5476 reconstruct.go:97] "Volume reconstruction finished" Nov 02 14:58:17.892617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.892550 5476 reconciler.go:29] "Reconciler: start to sync state" Nov 02 14:58:17.908257 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.908222 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 14:58:17.912333 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.912236 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 14:58:17.913065 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.912978 5476 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 14:58:17.914480 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.914454 5476 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 14:58:17.914480 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.914479 5476 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 14:58:17.914624 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.914501 5476 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 14:58:17.914624 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:17.914542 5476 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 14:58:17.921838 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.921794 5476 generic.go:356] "Generic (PLEG): container finished" podID="20771b11b53d11f8bd2b40993a0c6e2b" containerID="0b16f51cda23d2f1d2dce38caab4b591e190b72533e300c173887f2a0ca7afeb" exitCode=0 Nov 02 14:58:17.931065 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.931039 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 14:58:17.931404 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.931379 5476 generic.go:356] "Generic (PLEG): container finished" podID="449bb762-da7a-4a88-9796-1a1ef23e7c8c" containerID="de2e207e956519c5aa809e15a99d621e932641a49947107f7d6d4a805abe9061" exitCode=1 Nov 02 14:58:17.936622 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.936591 5476 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="52a3e3500c303e360343d669230281a4591cc1024cbd6a0aec2e047d9b59543c" exitCode=0 Nov 02 14:58:17.936622 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.936622 5476 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="24cee5b3b619e6260fe749e973764c3a09397404befb46177fcb7411fddf97c1" exitCode=0 Nov 02 14:58:17.936802 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.936632 5476 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="452828b8f34dbe35232fd3012cff8e4b90097712ecff30ac4bf5bf7522dc7a64" exitCode=0 Nov 02 14:58:17.936802 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.936639 5476 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="aa0969f7ada55f6b4530b74257fa5862536791ee04c7777ded259a18a439e50c" exitCode=0 Nov 02 14:58:17.936802 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.936646 5476 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="b41bf56725fab993146a4304c741f421719bd099bedecd52bc950a5be2ff09b7" exitCode=0 Nov 02 14:58:17.936802 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.936652 5476 generic.go:356] "Generic (PLEG): container finished" podID="0444057f-0080-4135-9d83-dca94858033e" containerID="40afaf4ef2bd07cf3eb5e4a8d4e052c098547430a16d75f8d2000b42c19276c1" exitCode=0 Nov 02 14:58:17.945154 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.945129 5476 generic.go:356] "Generic (PLEG): container finished" podID="ddab9016-d633-455c-a5e4-477bad67508d" containerID="72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe" exitCode=0 Nov 02 14:58:17.948445 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.948423 5476 generic.go:356] "Generic (PLEG): container finished" podID="d186e7b3-d2fa-4f5b-bd31-ee633608c8db" containerID="37e5f1844a97c4c134d019273ae6d0494d3523081bef1aedc7bc712b18b82c85" exitCode=0 Nov 02 14:58:17.964206 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.964179 5476 generic.go:356] "Generic (PLEG): container finished" podID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerID="8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9" exitCode=0 Nov 02 14:58:17.986753 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:17.986688 5476 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:58:17.991305 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:17.991278 5476 manager.go:324] Recovery completed Nov 02 14:58:18.012716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.012691 5476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:58:18.015051 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:18.015027 5476 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 14:58:18.016702 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.016679 5476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:58:18.016799 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.016717 5476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:58:18.016799 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.016732 5476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:58:18.019343 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019322 5476 cpu_manager.go:226] "Starting" policy="none" Nov 02 14:58:18.019343 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019338 5476 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 14:58:18.019496 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019344 5476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:58:18.019496 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019360 5476 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 14:58:18.019597 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019519 5476 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 14:58:18.019597 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019535 5476 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 14:58:18.019597 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019559 5476 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 14:58:18.019597 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019568 5476 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 14:58:18.019597 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019576 5476 policy_none.go:50] "Start" Nov 02 14:58:18.019597 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019589 5476 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 14:58:18.019864 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019602 5476 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 14:58:18.019864 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019792 5476 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 14:58:18.019864 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019803 5476 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 14:58:18.019864 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.019811 5476 policy_none.go:44] "Start" Nov 02 14:58:18.021599 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.021581 5476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:58:18.021681 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.021612 5476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:58:18.021681 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.021623 5476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:58:18.024612 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.024586 5476 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 14:58:18.027256 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.027243 5476 manager.go:341] "Starting Device Plugin manager" Nov 02 14:58:18.027342 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:18.027269 5476 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 14:58:18.027342 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.027279 5476 server.go:87] "Starting device plugin registration server" Nov 02 14:58:18.027553 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.027539 5476 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 14:58:18.027605 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.027554 5476 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 14:58:18.027687 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.027666 5476 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 14:58:18.027795 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.027760 5476 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 14:58:18.027795 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.027791 5476 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 14:58:18.027914 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.027798 5476 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 14:58:18.028756 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.027853 5476 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:58:18.027780558Z","UUID":"66225250-9a4f-47ef-aec9-3e77606cd088","Handler":null,"Name":"","Endpoint":""} Nov 02 14:58:18.029571 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:18.029549 5476 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 14:58:18.029658 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:18.029595 5476 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-141.us-west-2.compute.internal\" not found" Nov 02 14:58:18.030049 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.030033 5476 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 14:58:18.030128 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.030057 5476 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 14:58:18.062850 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.062823 5476 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 14:58:18.063004 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.062910 5476 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:58:18.066331 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.066309 5476 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:58:18.066432 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.066411 5476 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:58:18.127955 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.127925 5476 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:58:18.130380 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.130355 5476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:58:18.130546 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.130397 5476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:58:18.130546 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.130412 5476 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-141.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:58:18.132677 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.132633 5476 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.154898 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.154863 5476 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:58:18.215799 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.215751 5476 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal"] Nov 02 14:58:18.216487 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.216470 5476 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.216583 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.216507 5476 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.216622 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.216478 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" event={"ID":"5d1881377e4cd46a1e5cc5db6f951a84","Type":"ContainerStarted","Data":"bb7872ee1c8810b6836115e003a91b0dd5aee5c94142dbca144fa1d57aab4105"} Nov 02 14:58:18.216663 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.216623 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" event={"ID":"5d1881377e4cd46a1e5cc5db6f951a84","Type":"ContainerStarted","Data":"a75e8b0581b792e612a420f7c4bfdcb2db686b6af2d26b1ed651279f6dd9461a"} Nov 02 14:58:18.237043 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.236971 5476 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.237142 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.237049 5476 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.274234 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.274207 5476 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:58:18.274370 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:18.274251 5476 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.275312 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.275293 5476 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:58:18.275399 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:18.275339 5476 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.363713 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.363675 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d1881377e4cd46a1e5cc5db6f951a84-config\") pod \"kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"5d1881377e4cd46a1e5cc5db6f951a84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.363877 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.363726 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.363877 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.363743 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.464748 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.464718 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.464904 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.464757 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.464904 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.464781 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d1881377e4cd46a1e5cc5db6f951a84-config\") pod \"kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"5d1881377e4cd46a1e5cc5db6f951a84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.464904 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.464836 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5d1881377e4cd46a1e5cc5db6f951a84-config\") pod \"kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"5d1881377e4cd46a1e5cc5db6f951a84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.464904 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.464839 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.464904 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.464844 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/20771b11b53d11f8bd2b40993a0c6e2b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal\" (UID: \"20771b11b53d11f8bd2b40993a0c6e2b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:18.862961 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.862931 5476 apiserver.go:52] "Watching apiserver" Nov 02 14:58:18.975027 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:18.974949 5476 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:19.005881 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.005857 5476 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 14:58:19.006451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.006431 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-nrz94","openshift-ingress-canary/ingress-canary-vqgwt","openshift-monitoring/alertmanager-main-0","openshift-monitoring/node-exporter-qzm49","kube-system/global-pull-secret-syncer-vjfjv","openshift-dns/node-resolver-kjh9p","openshift-multus/multus-grr2c","kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-vxjb7","openshift-multus/network-metrics-daemon-f9zqb","kube-system/konnectivity-agent-xp4n5","openshift-monitoring/prometheus-operator-9cd89b74c-kwcld","openshift-multus/multus-additional-cni-plugins-lqtnw","openshift-dns/dns-default-qlm5f","openshift-network-operator/iptables-alerter-xsbm4","openshift-image-registry/node-ca-zdkx8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-141.us-west-2.compute.internal","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/telemeter-client-7899fb9795-f778g","openshift-ovn-kubernetes/ovnkube-node-qpkcp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7","openshift-insights/insights-runtime-extractor-lgnjt"] Nov 02 14:58:19.031023 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.030983 5476 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:58:19.031139 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:19.031045 5476 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal" Nov 02 14:58:19.067562 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067536 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 14:58:19.067562 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067536 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 14:58:19.067562 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067539 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 14:58:19.067794 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067538 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 14:58:19.067794 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067591 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 14:58:19.067794 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067622 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 14:58:19.067794 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067646 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 14:58:19.067794 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067653 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 14:58:19.067794 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067745 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 14:58:19.068026 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067842 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:58:19.068026 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067857 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 14:58:19.068026 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067879 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 14:58:19.068026 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.067883 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 14:58:19.068204 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.068086 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 14:58:19.068908 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.068893 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 14:58:19.068979 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.068895 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 14:58:19.068979 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.068915 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 14:58:19.069148 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069129 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:58:19.069215 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069134 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 14:58:19.069215 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069155 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 14:58:19.069215 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069165 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 14:58:19.069215 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069131 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 14:58:19.069215 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069160 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 14:58:19.069215 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069191 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 14:58:19.069215 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069159 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 14:58:19.069215 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069171 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 14:58:19.069519 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069500 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 14:58:19.069629 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069551 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 14:58:19.069629 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.069586 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 14:58:19.070497 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.070486 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 14:58:19.071381 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.071365 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 14:58:19.071532 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.071518 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 14:58:19.071587 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.071541 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 14:58:19.071587 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.071560 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 14:58:19.071672 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.071598 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 14:58:19.078953 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.078935 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 14:58:19.079434 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.079407 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 14:58:19.079625 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.079612 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 14:58:19.079805 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.079778 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 14:58:19.080064 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080038 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 14:58:19.080226 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080070 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 14:58:19.080226 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080081 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 14:58:19.080226 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080083 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rxzk9\"" Nov 02 14:58:19.080226 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080093 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 14:58:19.080226 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080083 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 14:58:19.080226 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080113 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 14:58:19.080804 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080787 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cr8rg\"" Nov 02 14:58:19.080912 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080824 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 14:58:19.080912 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080838 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 14:58:19.080912 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080860 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 14:58:19.080912 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.080877 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 14:58:19.089300 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.089284 5476 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 14:58:19.113869 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.113819 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 14:58:19.113951 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.113913 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2tgnv\"" Nov 02 14:58:19.113951 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.113928 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dlx4j\"" Nov 02 14:58:19.114174 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.114161 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 14:58:19.114291 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.114274 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 14:58:19.114340 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.114282 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ct4hh\"" Nov 02 14:58:19.114340 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.114303 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 14:58:19.114433 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.114413 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 14:58:19.114433 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.114426 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 14:58:19.114515 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.114503 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 14:58:19.114583 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.114516 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 14:58:19.114583 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.114519 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gmvcm\"" Nov 02 14:58:19.114583 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.114538 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 14:58:19.114716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.114607 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b9vbw\"" Nov 02 14:58:19.115164 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115143 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4crbl\"" Nov 02 14:58:19.115237 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115147 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4d7td\"" Nov 02 14:58:19.115237 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115181 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gwmlb\"" Nov 02 14:58:19.115237 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115185 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q62fl\"" Nov 02 14:58:19.115237 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115194 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tsn9p\"" Nov 02 14:58:19.115237 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115212 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 14:58:19.115451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115264 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-89ttf\"" Nov 02 14:58:19.115451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115268 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 14:58:19.115451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115296 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 14:58:19.115451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115311 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-29n45\"" Nov 02 14:58:19.115451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115326 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l9qfj\"" Nov 02 14:58:19.115451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115343 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zw4g4\"" Nov 02 14:58:19.115451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115356 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 14:58:19.115451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115350 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 14:58:19.115451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115380 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s2tzl\"" Nov 02 14:58:19.115451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115406 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c2bv5\"" Nov 02 14:58:19.115451 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.115409 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hnsn9\"" Nov 02 14:58:19.126196 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.126175 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 14:58:19.126262 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.126212 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 14:58:19.126338 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.126298 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 14:58:19.127174 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.127152 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 14:58:19.127243 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.127231 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 14:58:19.159196 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.159176 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 14:58:19.159779 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.159764 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 14:58:19.160520 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.160501 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 14:58:19.160617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.160538 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 14:58:19.160617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.160582 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 14:58:19.160729 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.160685 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 14:58:19.160729 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.160698 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 14:58:19.160882 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.160736 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 14:58:19.160882 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.160760 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cp3c23k1g144d\"" Nov 02 14:58:19.160882 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.160779 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-89xs6\"" Nov 02 14:58:19.161342 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.161320 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 14:58:19.161476 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.161460 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 14:58:19.169110 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169089 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.169208 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169120 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-modprobe-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.169208 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169137 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efc5a077-d018-4792-8551-b9c3594967dd-agent-certs\") pod \"konnectivity-agent-xp4n5\" (UID: \"efc5a077-d018-4792-8551-b9c3594967dd\") " pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:58:19.169208 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169152 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-config\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.169208 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169169 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.169396 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169234 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.169396 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169365 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-socket-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.169497 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169402 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-systemd-units\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.169497 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169434 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-serving-certs-ca-bundle\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.169497 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169470 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.169624 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169494 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.169624 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169511 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efc5a077-d018-4792-8551-b9c3594967dd-agent-certs\") pod \"konnectivity-agent-xp4n5\" (UID: \"efc5a077-d018-4792-8551-b9c3594967dd\") " pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:58:19.169624 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169511 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.169712 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169623 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.169712 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169629 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.169712 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169692 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.169712 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169706 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-serving-certs-ca-bundle\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.169869 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169725 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.169869 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169751 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-var-lib-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.169869 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169781 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.169869 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169846 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-config\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.170045 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169884 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-multus\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.170045 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169917 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.170045 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.169944 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.170130 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170044 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29cc4\" (UniqueName: \"kubernetes.io/projected/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-kube-api-access-29cc4\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:58:19.170171 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170129 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.170203 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170173 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.170238 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170199 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.170275 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170250 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.170322 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170283 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-etc-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.170429 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170332 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-hostroot\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.170429 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170382 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-multus-certs\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.170504 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170437 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcklx\" (UniqueName: \"kubernetes.io/projected/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-api-access-wcklx\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:58:19.170504 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170444 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.170504 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170467 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-etc-tuned\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.170504 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170491 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-tmp\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.170699 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170514 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-cnibin\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.170699 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170538 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d8b64c-559c-4806-8599-25b62c630fb4-tmp-dir\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:58:19.170699 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170561 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-device-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.170699 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170578 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-etc-tuned\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.170699 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170586 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8880c66c-32b2-4a81-a174-34242093784d-tmp\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.170699 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170586 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j698s\" (UniqueName: \"kubernetes.io/projected/2e7e4e7b-c190-420d-b438-b2c9e1074795-kube-api-access-j698s\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:58:19.170699 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170514 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170726 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25f92d33-6475-49f1-bfff-691a5dec0ae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170781 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170782 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d8b64c-559c-4806-8599-25b62c630fb4-tmp-dir\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170830 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-var-lib-kubelet\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170855 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66740b77-b82e-49c2-b8d0-af48d25147cb-cert\") pod \"ingress-canary-vqgwt\" (UID: \"66740b77-b82e-49c2-b8d0-af48d25147cb\") " pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170879 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170910 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170940 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnvct\" (UniqueName: \"kubernetes.io/projected/b3d8b64c-559c-4806-8599-25b62c630fb4-kube-api-access-lnvct\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170961 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/25f92d33-6475-49f1-bfff-691a5dec0ae0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.170974 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-netns\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171013 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-run\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171040 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.171073 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171068 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171070 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171111 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-federate-client-tls\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171126 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171137 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysconfig\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171146 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66740b77-b82e-49c2-b8d0-af48d25147cb-cert\") pod \"ingress-canary-vqgwt\" (UID: \"66740b77-b82e-49c2-b8d0-af48d25147cb\") " pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171170 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbr8c\" (UniqueName: \"kubernetes.io/projected/0444057f-0080-4135-9d83-dca94858033e-kube-api-access-wbr8c\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171210 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171216 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171221 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4bfe09e4-8fce-4149-a9fb-0a222124f121-original-pull-secret\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171288 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171386 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-federate-client-tls\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171394 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171420 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171485 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-kubernetes\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171547 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-sys\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171578 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171577 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171582 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.171630 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171638 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr7rp\" (UniqueName: \"kubernetes.io/projected/b5cc5088-ab54-47e3-bcca-7cb792f88880-kube-api-access-lr7rp\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171612 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171662 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-metrics-client-ca\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171688 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-lib-modules\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171725 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171752 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-root\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171768 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171792 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-textfile\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171809 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e8592fc-f083-4eb8-b934-517c47f123c4-metrics-client-ca\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171907 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-tls\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171913 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171944 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-textfile\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.171978 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172027 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2slcd\" (UniqueName: \"kubernetes.io/projected/66740b77-b82e-49c2-b8d0-af48d25147cb-kube-api-access-2slcd\") pod \"ingress-canary-vqgwt\" (UID: \"66740b77-b82e-49c2-b8d0-af48d25147cb\") " pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172053 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989be356-90d5-4c62-8200-5d5a8d2af857-tmp-dir\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172069 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsfph\" (UniqueName: \"kubernetes.io/projected/989be356-90d5-4c62-8200-5d5a8d2af857-kube-api-access-jsfph\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172114 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172143 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/989be356-90d5-4c62-8200-5d5a8d2af857-tmp-dir\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172224 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172232 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172293 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172328 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-tls\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172339 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172374 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-node-log\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172391 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovn-node-metrics-cert\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172407 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hh7\" (UniqueName: \"kubernetes.io/projected/449bb762-da7a-4a88-9796-1a1ef23e7c8c-kube-api-access-x5hh7\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.172441 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172425 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smgsz\" (UniqueName: \"kubernetes.io/projected/d1af9240-8442-4911-b1d9-5570aea28abf-kube-api-access-smgsz\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172461 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-conf\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172499 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-system-cni-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172528 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-kubelet\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172556 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25f92d33-6475-49f1-bfff-691a5dec0ae0-data-volume\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172562 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172583 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172621 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172675 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/25f92d33-6475-49f1-bfff-691a5dec0ae0-data-volume\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172683 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172728 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172761 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovn-node-metrics-cert\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172791 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172816 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172856 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-log-socket\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172919 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.172940 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173019 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173096 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173111 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173122 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173150 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-wtmp\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173177 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-out\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173212 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-script-lib\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173285 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-etc-kubernetes\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173298 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0444057f-0080-4135-9d83-dca94858033e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173291 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-out\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173340 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-client-tls\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173368 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-kubelet\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173394 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-conf-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173418 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-dbus\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173445 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173459 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-script-lib\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173489 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3d8b64c-559c-4806-8599-25b62c630fb4-hosts-file\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173521 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173524 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-telemeter-client-tls\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.173569 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173571 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efc5a077-d018-4792-8551-b9c3594967dd-konnectivity-ca\") pod \"konnectivity-agent-xp4n5\" (UID: \"efc5a077-d018-4792-8551-b9c3594967dd\") " pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173676 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173708 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173733 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-netns\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173759 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-web-config\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173776 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e8592fc-f083-4eb8-b934-517c47f123c4-secret-telemeter-client\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173816 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25f92d33-6475-49f1-bfff-691a5dec0ae0-crio-socket\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173810 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efc5a077-d018-4792-8551-b9c3594967dd-konnectivity-ca\") pod \"konnectivity-agent-xp4n5\" (UID: \"efc5a077-d018-4792-8551-b9c3594967dd\") " pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173844 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173863 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173878 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-kubelet-config\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173907 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-accelerators-collector-config\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.173964 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-web-config\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174052 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-volume\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174097 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-slash\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174110 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-metrics-certs\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174116 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-systemd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174155 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cni-binary-copy\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174205 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-accelerators-collector-config\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174236 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174268 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ffbd00b-29ae-4d63-97b8-edf990224dcd-host-slash\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174272 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-volume\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174285 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-host\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174294 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174307 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174324 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/989be356-90d5-4c62-8200-5d5a8d2af857-config-volume\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174350 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cni-binary-copy\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174354 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-ovn\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174398 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174433 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz25j\" (UniqueName: \"kubernetes.io/projected/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-kube-api-access-hz25j\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174461 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zjwc\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-kube-api-access-8zjwc\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174508 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174530 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/989be356-90d5-4c62-8200-5d5a8d2af857-config-volume\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174542 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174513 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174573 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174604 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-bin\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174606 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174621 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-k8s-cni-cncf-io\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174646 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ffbd00b-29ae-4d63-97b8-edf990224dcd-iptables-alerter-script\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174683 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1af9240-8442-4911-b1d9-5570aea28abf-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174708 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174713 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174739 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p65ch\" (UniqueName: \"kubernetes.io/projected/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-kube-api-access-p65ch\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174760 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174792 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-sys-fs\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174817 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174826 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ffbd00b-29ae-4d63-97b8-edf990224dcd-iptables-alerter-script\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174840 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-bin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174844 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1af9240-8442-4911-b1d9-5570aea28abf-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174907 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174940 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e7e4e7b-c190-420d-b438-b2c9e1074795-host\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.174974 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hzs7\" (UniqueName: \"kubernetes.io/projected/4ffbd00b-29ae-4d63-97b8-edf990224dcd-kube-api-access-7hzs7\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175026 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-systemd\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175037 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175056 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/989be356-90d5-4c62-8200-5d5a8d2af857-metrics-tls\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175090 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.175094 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175123 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175150 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4p4n\" (UniqueName: \"kubernetes.io/projected/8880c66c-32b2-4a81-a174-34242093784d-kube-api-access-r4p4n\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175185 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxp4\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-kube-api-access-vsxp4\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175199 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/989be356-90d5-4c62-8200-5d5a8d2af857-metrics-tls\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175213 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-registration-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175246 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-netd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175307 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-config\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175341 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-socket-dir-parent\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175346 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d1af9240-8442-4911-b1d9-5570aea28abf-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175367 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175396 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cnibin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175416 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmc4m\" (UniqueName: \"kubernetes.io/projected/5e8592fc-f083-4eb8-b934-517c47f123c4-kube-api-access-fmc4m\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175441 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-env-overrides\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175480 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e7e4e7b-c190-420d-b438-b2c9e1074795-serviceca\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175501 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-ovnkube-config\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175505 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-metrics-client-ca\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175551 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-sys\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175580 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-system-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175594 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/449bb762-da7a-4a88-9796-1a1ef23e7c8c-env-overrides\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175608 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-daemon-config\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175609 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175687 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-metrics-client-ca\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.175792 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-daemon-config\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.176986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.176125 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e7e4e7b-c190-420d-b438-b2c9e1074795-serviceca\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:58:19.226986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.226955 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29cc4\" (UniqueName: \"kubernetes.io/projected/13823b41-3da3-4a0b-bf1d-4c9d1b24608f-kube-api-access-29cc4\") pod \"network-metrics-daemon-f9zqb\" (UID: \"13823b41-3da3-4a0b-bf1d-4c9d1b24608f\") " pod="openshift-multus/network-metrics-daemon-f9zqb" Nov 02 14:58:19.231770 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.231746 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcklx\" (UniqueName: \"kubernetes.io/projected/25f92d33-6475-49f1-bfff-691a5dec0ae0-kube-api-access-wcklx\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:58:19.231965 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.231944 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j698s\" (UniqueName: \"kubernetes.io/projected/2e7e4e7b-c190-420d-b438-b2c9e1074795-kube-api-access-j698s\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:58:19.232026 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.231949 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr7rp\" (UniqueName: \"kubernetes.io/projected/b5cc5088-ab54-47e3-bcca-7cb792f88880-kube-api-access-lr7rp\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.237014 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.236975 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p65ch\" (UniqueName: \"kubernetes.io/projected/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-kube-api-access-p65ch\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.237099 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.237033 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smgsz\" (UniqueName: \"kubernetes.io/projected/d1af9240-8442-4911-b1d9-5570aea28abf-kube-api-access-smgsz\") pod \"prometheus-operator-9cd89b74c-kwcld\" (UID: \"d1af9240-8442-4911-b1d9-5570aea28abf\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kwcld" Nov 02 14:58:19.242114 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.242090 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnvct\" (UniqueName: \"kubernetes.io/projected/b3d8b64c-559c-4806-8599-25b62c630fb4-kube-api-access-lnvct\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:58:19.242215 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.242182 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hzs7\" (UniqueName: \"kubernetes.io/projected/4ffbd00b-29ae-4d63-97b8-edf990224dcd-kube-api-access-7hzs7\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:58:19.248566 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.248504 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz25j\" (UniqueName: \"kubernetes.io/projected/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-kube-api-access-hz25j\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.248660 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.248575 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbr8c\" (UniqueName: \"kubernetes.io/projected/0444057f-0080-4135-9d83-dca94858033e-kube-api-access-wbr8c\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.248660 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.248620 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsfph\" (UniqueName: \"kubernetes.io/projected/989be356-90d5-4c62-8200-5d5a8d2af857-kube-api-access-jsfph\") pod \"dns-default-qlm5f\" (UID: \"989be356-90d5-4c62-8200-5d5a8d2af857\") " pod="openshift-dns/dns-default-qlm5f" Nov 02 14:58:19.248779 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.248681 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hh7\" (UniqueName: \"kubernetes.io/projected/449bb762-da7a-4a88-9796-1a1ef23e7c8c-kube-api-access-x5hh7\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.248779 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.248762 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxp4\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-kube-api-access-vsxp4\") pod \"prometheus-k8s-0\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:19.248848 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.248807 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmc4m\" (UniqueName: \"kubernetes.io/projected/5e8592fc-f083-4eb8-b934-517c47f123c4-kube-api-access-fmc4m\") pod \"telemeter-client-7899fb9795-f778g\" (UID: \"5e8592fc-f083-4eb8-b934-517c47f123c4\") " pod="openshift-monitoring/telemeter-client-7899fb9795-f778g" Nov 02 14:58:19.248848 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.248828 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mjmh\" (UniqueName: \"kubernetes.io/projected/d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9-kube-api-access-6mjmh\") pod \"network-check-target-vxjb7\" (UID: \"d3427e5f-57ef-48d9-ac0f-2b573fb2e8d9\") " pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:58:19.248971 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.248954 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2slcd\" (UniqueName: \"kubernetes.io/projected/66740b77-b82e-49c2-b8d0-af48d25147cb-kube-api-access-2slcd\") pod \"ingress-canary-vqgwt\" (UID: \"66740b77-b82e-49c2-b8d0-af48d25147cb\") " pod="openshift-ingress-canary/ingress-canary-vqgwt" Nov 02 14:58:19.249035 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.249021 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4p4n\" (UniqueName: \"kubernetes.io/projected/8880c66c-32b2-4a81-a174-34242093784d-kube-api-access-r4p4n\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.253301 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.253285 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zjwc\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-kube-api-access-8zjwc\") pod \"alertmanager-main-0\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:19.276476 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276457 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.276567 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276482 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-netns\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.276567 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276499 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25f92d33-6475-49f1-bfff-691a5dec0ae0-crio-socket\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:58:19.276567 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276514 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-kubelet-config\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:58:19.276567 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276531 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-slash\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.276567 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276566 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-slash\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276573 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-netns\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276598 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276609 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-systemd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276611 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-kubelet-config\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276613 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/25f92d33-6475-49f1-bfff-691a5dec0ae0-crio-socket\") pod \"insights-runtime-extractor-lgnjt\" (UID: \"25f92d33-6475-49f1-bfff-691a5dec0ae0\") " pod="openshift-insights/insights-runtime-extractor-lgnjt" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276630 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ffbd00b-29ae-4d63-97b8-edf990224dcd-host-slash\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276653 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-host\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276668 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-systemd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276672 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276695 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-ovn\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276708 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-host\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276718 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.276758 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276753 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-ovn\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276774 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276820 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ffbd00b-29ae-4d63-97b8-edf990224dcd-host-slash\") pod \"iptables-alerter-xsbm4\" (UID: \"4ffbd00b-29ae-4d63-97b8-edf990224dcd\") " pod="openshift-network-operator/iptables-alerter-xsbm4" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276832 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276848 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276865 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-run-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276876 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-bin\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276902 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-k8s-cni-cncf-io\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276928 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-bin\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276931 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-sys-fs\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276955 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276958 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-k8s-cni-cncf-io\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276979 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-bin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.276987 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-sys-fs\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277025 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e7e4e7b-c190-420d-b438-b2c9e1074795-host\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277032 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277037 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-bin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277051 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-systemd\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277071 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e7e4e7b-c190-420d-b438-b2c9e1074795-host\") pod \"node-ca-zdkx8\" (UID: \"2e7e4e7b-c190-420d-b438-b2c9e1074795\") " pod="openshift-image-registry/node-ca-zdkx8" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277079 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277118 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-systemd\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277135 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277138 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-registration-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277160 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-registration-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277175 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-netd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277202 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-socket-dir-parent\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277221 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-cni-netd\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277226 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cnibin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277260 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-socket-dir-parent\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277261 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-sys\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277286 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-system-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277300 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-cnibin\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277311 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-modprobe-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277329 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-sys\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277335 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-system-cni-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277340 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-socket-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277363 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-systemd-units\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277369 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-socket-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277388 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277397 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-modprobe-d\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277402 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-systemd-units\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277410 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277429 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.277400 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277431 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-var-lib-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277455 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-var-lib-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277454 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-multus\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277464 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277481 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-etc-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277495 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-hostroot\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277510 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-multus-certs\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277526 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-cnibin\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277539 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-device-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277558 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-var-lib-kubelet\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277569 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-etc-openvswitch\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277493 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-cni-multus\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277576 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-cnibin\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277574 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-netns\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277604 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5cc5088-ab54-47e3-bcca-7cb792f88880-device-dir\") pod \"aws-ebs-csi-driver-node-lwsj7\" (UID: \"b5cc5088-ab54-47e3-bcca-7cb792f88880\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwsj7" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277608 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-run-netns\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277610 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-hostroot\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277625 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-var-lib-kubelet\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277613 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-run\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277645 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-run\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277655 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysconfig\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277675 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-run-multus-certs\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277676 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-kubernetes\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277697 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-sys\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277711 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysconfig\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277712 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-lib-modules\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277748 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-root\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277758 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-kubernetes\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277776 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-sys\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277787 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-lib-modules\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277786 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-node-log\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277797 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-root\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277812 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-node-log\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277820 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-conf\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277839 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-system-cni-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277855 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-kubelet\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277875 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-log-socket\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277900 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0444057f-0080-4135-9d83-dca94858033e-system-cni-dir\") pod \"multus-additional-cni-plugins-lqtnw\" (UID: \"0444057f-0080-4135-9d83-dca94858033e\") " pod="openshift-multus/multus-additional-cni-plugins-lqtnw" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277936 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-wtmp\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277934 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-log-socket\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277902 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d186e7b3-d2fa-4f5b-bd31-ee633608c8db-node-exporter-wtmp\") pod \"node-exporter-qzm49\" (UID: \"d186e7b3-d2fa-4f5b-bd31-ee633608c8db\") " pod="openshift-monitoring/node-exporter-qzm49" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277969 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8880c66c-32b2-4a81-a174-34242093784d-etc-sysctl-conf\") pod \"tuned-nrz94\" (UID: \"8880c66c-32b2-4a81-a174-34242093784d\") " pod="openshift-cluster-node-tuning-operator/tuned-nrz94" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277971 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-etc-kubernetes\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277939 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-host-var-lib-kubelet\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.277988 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-etc-kubernetes\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.278029 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-kubelet\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.278058 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-conf-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.278081 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-dbus\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.278100 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/449bb762-da7a-4a88-9796-1a1ef23e7c8c-host-kubelet\") pod \"ovnkube-node-qpkcp\" (UID: \"449bb762-da7a-4a88-9796-1a1ef23e7c8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.278105 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3d8b64c-559c-4806-8599-25b62c630fb4-hosts-file\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.278140 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752-multus-conf-dir\") pod \"multus-grr2c\" (UID: \"fb2a2b02-cc3e-4a70-9d4e-a7c2bf896752\") " pod="openshift-multus/multus-grr2c" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.278175 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4bfe09e4-8fce-4149-a9fb-0a222124f121-dbus\") pod \"global-pull-secret-syncer-vjfjv\" (UID: \"4bfe09e4-8fce-4149-a9fb-0a222124f121\") " pod="kube-system/global-pull-secret-syncer-vjfjv" Nov 02 14:58:19.278807 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.278190 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b3d8b64c-559c-4806-8599-25b62c630fb4-hosts-file\") pod \"node-resolver-kjh9p\" (UID: \"b3d8b64c-559c-4806-8599-25b62c630fb4\") " pod="openshift-dns/node-resolver-kjh9p" Nov 02 14:58:19.320145 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.320126 5476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:58:19.711036 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:19.710984 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qlm5f" Nov 02 14:58:20.146674 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:20.146640 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 14:58:20.560623 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:20.560541 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xp4n5" Nov 02 14:58:20.595744 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:20.595717 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:21.355749 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:21.355710 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:21.374224 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:21.374193 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:21.988507 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:21.988478 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerStarted","Data":"2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2"} Nov 02 14:58:22.735219 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:22.735188 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:58:22.995749 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:22.995665 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerStarted","Data":"e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c"} Nov 02 14:58:22.995749 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:22.995735 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerStarted","Data":"3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0"} Nov 02 14:58:22.995956 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:22.995772 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerStarted","Data":"5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1"} Nov 02 14:58:26.700959 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:26.700926 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:26.701380 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:26.701207 5476 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:58:28.880608 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:28.880581 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qpkcp" Nov 02 14:58:37.016692 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.016657 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 14:58:37.017173 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.017041 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="alertmanager" containerID="cri-o://686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497" gracePeriod=120 Nov 02 14:58:37.017173 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.017053 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy-metric" containerID="cri-o://7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841" gracePeriod=120 Nov 02 14:58:37.017173 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.017114 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="config-reloader" containerID="cri-o://5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a" gracePeriod=120 Nov 02 14:58:37.017356 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.017148 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="prom-label-proxy" containerID="cri-o://60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b" gracePeriod=120 Nov 02 14:58:37.017356 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.017100 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy-web" containerID="cri-o://4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4" gracePeriod=120 Nov 02 14:58:37.017356 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.017117 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy" containerID="cri-o://27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa" gracePeriod=120 Nov 02 14:58:37.266279 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.266248 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:37.319930 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.319831 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-trusted-ca-bundle\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.319930 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.319889 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-main-tls\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.319930 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.319925 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-web-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-web-config\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.320208 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.319974 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-cluster-tls-config\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.320208 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320038 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-out\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.320208 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320080 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.320208 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320116 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-volume\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-volume\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.320208 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320146 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-metrics-client-ca\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.320208 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320180 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.320515 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320240 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-web\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.320515 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320282 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-main-db\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.320515 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320310 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-kube-api-access-8zjwc\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-kube-api-access-8zjwc\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.320515 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320331 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-trusted-ca-bundle" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:37.320515 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320345 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-tls-assets\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-tls-assets\") pod \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\" (UID: \"c8b78c94-8df4-49a1-b365-bc0b92eae8ba\") " Nov 02 14:58:37.320778 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320557 5476 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.320778 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.320652 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-metrics-client-ca" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:37.321240 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.321210 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-main-db" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:37.323491 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.323446 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-main-tls" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:37.323831 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.323806 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-web" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:37.323926 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.323869 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-tls-assets" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:37.324584 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.324559 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-metric" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:37.324831 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.324805 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-volume" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:37.325195 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.325160 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-out" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:37.325754 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.325728 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-kube-api-access-8zjwc" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "kube-api-access-8zjwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:37.325896 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.325867 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:37.328250 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.328229 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-cluster-tls-config" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:37.334408 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.334382 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-web-config" pod "c8b78c94-8df4-49a1-b365-bc0b92eae8ba" (UID: "c8b78c94-8df4-49a1-b365-bc0b92eae8ba"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:37.421956 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.421917 5476 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.422129 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.421958 5476 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.422129 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.421980 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-metrics-client-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.422129 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.422023 5476 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.422129 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.422042 5476 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.422129 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.422057 5476 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-alertmanager-main-db\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.422129 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.422071 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zjwc\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-kube-api-access-8zjwc\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.422129 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.422086 5476 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-tls-assets\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.422129 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.422102 5476 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-secret-alertmanager-main-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.422129 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.422119 5476 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-web-config\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.422393 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.422136 5476 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-cluster-tls-config\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:37.422393 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:37.422151 5476 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8b78c94-8df4-49a1-b365-bc0b92eae8ba-config-out\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:38.053251 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053209 5476 generic.go:356] "Generic (PLEG): container finished" podID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerID="60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b" exitCode=0 Nov 02 14:58:38.053251 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053243 5476 generic.go:356] "Generic (PLEG): container finished" podID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerID="7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841" exitCode=0 Nov 02 14:58:38.053251 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053253 5476 generic.go:356] "Generic (PLEG): container finished" podID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerID="27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa" exitCode=0 Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053247 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerDied","Data":"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b"} Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053308 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053261 5476 generic.go:356] "Generic (PLEG): container finished" podID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerID="4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4" exitCode=0 Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053355 5476 generic.go:356] "Generic (PLEG): container finished" podID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerID="5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a" exitCode=0 Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053368 5476 generic.go:356] "Generic (PLEG): container finished" podID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerID="686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497" exitCode=0 Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053369 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerDied","Data":"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841"} Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053411 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerDied","Data":"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa"} Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053451 5476 scope.go:122] "RemoveContainer" containerID="60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b" Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053453 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerDied","Data":"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4"} Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053589 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerDied","Data":"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a"} Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053637 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerDied","Data":"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497"} Nov 02 14:58:38.053774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.053682 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c8b78c94-8df4-49a1-b365-bc0b92eae8ba","Type":"ContainerDied","Data":"5015acc47cf4b5020f7b0f0e4faf6d8dc1c63d0b7daa01fa7708869ebc020ac6"} Nov 02 14:58:38.061097 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.061075 5476 scope.go:122] "RemoveContainer" containerID="7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841" Nov 02 14:58:38.067834 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.067815 5476 scope.go:122] "RemoveContainer" containerID="27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa" Nov 02 14:58:38.074682 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.074651 5476 scope.go:122] "RemoveContainer" containerID="4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4" Nov 02 14:58:38.081162 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.081144 5476 scope.go:122] "RemoveContainer" containerID="5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a" Nov 02 14:58:38.089565 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.089541 5476 scope.go:122] "RemoveContainer" containerID="686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497" Nov 02 14:58:38.096609 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.096591 5476 scope.go:122] "RemoveContainer" containerID="8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9" Nov 02 14:58:38.103621 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.103596 5476 scope.go:122] "RemoveContainer" containerID="60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b" Nov 02 14:58:38.103924 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:38.103897 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": container with ID starting with 60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b not found: ID does not exist" containerID="60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b" Nov 02 14:58:38.104025 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.103933 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b"} err="failed to get container status \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": rpc error: code = NotFound desc = could not find container \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": container with ID starting with 60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b not found: ID does not exist" Nov 02 14:58:38.104025 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.103969 5476 scope.go:122] "RemoveContainer" containerID="7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841" Nov 02 14:58:38.104247 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:38.104228 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": container with ID starting with 7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841 not found: ID does not exist" containerID="7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841" Nov 02 14:58:38.104292 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.104253 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841"} err="failed to get container status \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": rpc error: code = NotFound desc = could not find container \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": container with ID starting with 7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841 not found: ID does not exist" Nov 02 14:58:38.104292 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.104268 5476 scope.go:122] "RemoveContainer" containerID="27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa" Nov 02 14:58:38.104500 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:38.104475 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": container with ID starting with 27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa not found: ID does not exist" containerID="27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa" Nov 02 14:58:38.104560 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.104510 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa"} err="failed to get container status \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": rpc error: code = NotFound desc = could not find container \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": container with ID starting with 27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa not found: ID does not exist" Nov 02 14:58:38.104560 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.104534 5476 scope.go:122] "RemoveContainer" containerID="4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4" Nov 02 14:58:38.104756 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:38.104737 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": container with ID starting with 4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4 not found: ID does not exist" containerID="4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4" Nov 02 14:58:38.104801 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.104761 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4"} err="failed to get container status \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": rpc error: code = NotFound desc = could not find container \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": container with ID starting with 4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4 not found: ID does not exist" Nov 02 14:58:38.104801 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.104776 5476 scope.go:122] "RemoveContainer" containerID="5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a" Nov 02 14:58:38.105014 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:38.104978 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": container with ID starting with 5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a not found: ID does not exist" containerID="5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a" Nov 02 14:58:38.105060 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.105030 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a"} err="failed to get container status \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": rpc error: code = NotFound desc = could not find container \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": container with ID starting with 5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a not found: ID does not exist" Nov 02 14:58:38.105060 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.105044 5476 scope.go:122] "RemoveContainer" containerID="686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497" Nov 02 14:58:38.105264 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:38.105248 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": container with ID starting with 686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497 not found: ID does not exist" containerID="686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497" Nov 02 14:58:38.105311 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.105269 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497"} err="failed to get container status \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": rpc error: code = NotFound desc = could not find container \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": container with ID starting with 686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497 not found: ID does not exist" Nov 02 14:58:38.105311 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.105281 5476 scope.go:122] "RemoveContainer" containerID="8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9" Nov 02 14:58:38.105485 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:38.105467 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": container with ID starting with 8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9 not found: ID does not exist" containerID="8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9" Nov 02 14:58:38.105553 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.105492 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9"} err="failed to get container status \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": rpc error: code = NotFound desc = could not find container \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": container with ID starting with 8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9 not found: ID does not exist" Nov 02 14:58:38.105553 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.105510 5476 scope.go:122] "RemoveContainer" containerID="60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b" Nov 02 14:58:38.105729 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.105714 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b"} err="failed to get container status \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": rpc error: code = NotFound desc = could not find container \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": container with ID starting with 60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b not found: ID does not exist" Nov 02 14:58:38.105781 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.105729 5476 scope.go:122] "RemoveContainer" containerID="7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841" Nov 02 14:58:38.105923 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.105906 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841"} err="failed to get container status \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": rpc error: code = NotFound desc = could not find container \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": container with ID starting with 7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841 not found: ID does not exist" Nov 02 14:58:38.105923 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.105922 5476 scope.go:122] "RemoveContainer" containerID="27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa" Nov 02 14:58:38.106171 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.106153 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa"} err="failed to get container status \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": rpc error: code = NotFound desc = could not find container \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": container with ID starting with 27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa not found: ID does not exist" Nov 02 14:58:38.106223 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.106171 5476 scope.go:122] "RemoveContainer" containerID="4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4" Nov 02 14:58:38.106377 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.106359 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4"} err="failed to get container status \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": rpc error: code = NotFound desc = could not find container \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": container with ID starting with 4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4 not found: ID does not exist" Nov 02 14:58:38.106418 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.106376 5476 scope.go:122] "RemoveContainer" containerID="5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a" Nov 02 14:58:38.106570 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.106551 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a"} err="failed to get container status \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": rpc error: code = NotFound desc = could not find container \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": container with ID starting with 5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a not found: ID does not exist" Nov 02 14:58:38.106570 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.106567 5476 scope.go:122] "RemoveContainer" containerID="686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497" Nov 02 14:58:38.106810 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.106790 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497"} err="failed to get container status \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": rpc error: code = NotFound desc = could not find container \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": container with ID starting with 686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497 not found: ID does not exist" Nov 02 14:58:38.106861 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.106811 5476 scope.go:122] "RemoveContainer" containerID="8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9" Nov 02 14:58:38.107056 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.107035 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9"} err="failed to get container status \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": rpc error: code = NotFound desc = could not find container \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": container with ID starting with 8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9 not found: ID does not exist" Nov 02 14:58:38.107095 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.107057 5476 scope.go:122] "RemoveContainer" containerID="60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b" Nov 02 14:58:38.107304 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.107275 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b"} err="failed to get container status \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": rpc error: code = NotFound desc = could not find container \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": container with ID starting with 60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b not found: ID does not exist" Nov 02 14:58:38.107375 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.107303 5476 scope.go:122] "RemoveContainer" containerID="7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841" Nov 02 14:58:38.107530 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.107514 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841"} err="failed to get container status \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": rpc error: code = NotFound desc = could not find container \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": container with ID starting with 7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841 not found: ID does not exist" Nov 02 14:58:38.107576 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.107531 5476 scope.go:122] "RemoveContainer" containerID="27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa" Nov 02 14:58:38.107732 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.107714 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa"} err="failed to get container status \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": rpc error: code = NotFound desc = could not find container \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": container with ID starting with 27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa not found: ID does not exist" Nov 02 14:58:38.107774 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.107731 5476 scope.go:122] "RemoveContainer" containerID="4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4" Nov 02 14:58:38.107919 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.107901 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4"} err="failed to get container status \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": rpc error: code = NotFound desc = could not find container \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": container with ID starting with 4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4 not found: ID does not exist" Nov 02 14:58:38.107988 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.107919 5476 scope.go:122] "RemoveContainer" containerID="5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a" Nov 02 14:58:38.108156 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.108136 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a"} err="failed to get container status \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": rpc error: code = NotFound desc = could not find container \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": container with ID starting with 5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a not found: ID does not exist" Nov 02 14:58:38.108156 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.108154 5476 scope.go:122] "RemoveContainer" containerID="686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497" Nov 02 14:58:38.108360 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.108344 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497"} err="failed to get container status \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": rpc error: code = NotFound desc = could not find container \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": container with ID starting with 686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497 not found: ID does not exist" Nov 02 14:58:38.108360 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.108360 5476 scope.go:122] "RemoveContainer" containerID="8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9" Nov 02 14:58:38.108561 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.108545 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9"} err="failed to get container status \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": rpc error: code = NotFound desc = could not find container \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": container with ID starting with 8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9 not found: ID does not exist" Nov 02 14:58:38.108605 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.108561 5476 scope.go:122] "RemoveContainer" containerID="60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b" Nov 02 14:58:38.108735 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.108719 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b"} err="failed to get container status \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": rpc error: code = NotFound desc = could not find container \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": container with ID starting with 60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b not found: ID does not exist" Nov 02 14:58:38.108773 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.108734 5476 scope.go:122] "RemoveContainer" containerID="7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841" Nov 02 14:58:38.108920 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.108904 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841"} err="failed to get container status \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": rpc error: code = NotFound desc = could not find container \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": container with ID starting with 7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841 not found: ID does not exist" Nov 02 14:58:38.108960 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.108919 5476 scope.go:122] "RemoveContainer" containerID="27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa" Nov 02 14:58:38.109117 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.109098 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa"} err="failed to get container status \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": rpc error: code = NotFound desc = could not find container \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": container with ID starting with 27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa not found: ID does not exist" Nov 02 14:58:38.109198 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.109118 5476 scope.go:122] "RemoveContainer" containerID="4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4" Nov 02 14:58:38.109316 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.109299 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4"} err="failed to get container status \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": rpc error: code = NotFound desc = could not find container \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": container with ID starting with 4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4 not found: ID does not exist" Nov 02 14:58:38.109316 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.109315 5476 scope.go:122] "RemoveContainer" containerID="5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a" Nov 02 14:58:38.109503 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.109486 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a"} err="failed to get container status \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": rpc error: code = NotFound desc = could not find container \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": container with ID starting with 5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a not found: ID does not exist" Nov 02 14:58:38.109545 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.109503 5476 scope.go:122] "RemoveContainer" containerID="686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497" Nov 02 14:58:38.109685 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.109669 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497"} err="failed to get container status \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": rpc error: code = NotFound desc = could not find container \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": container with ID starting with 686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497 not found: ID does not exist" Nov 02 14:58:38.109685 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.109684 5476 scope.go:122] "RemoveContainer" containerID="8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9" Nov 02 14:58:38.109859 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.109843 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9"} err="failed to get container status \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": rpc error: code = NotFound desc = could not find container \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": container with ID starting with 8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9 not found: ID does not exist" Nov 02 14:58:38.109859 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.109858 5476 scope.go:122] "RemoveContainer" containerID="60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b" Nov 02 14:58:38.110049 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110033 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b"} err="failed to get container status \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": rpc error: code = NotFound desc = could not find container \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": container with ID starting with 60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b not found: ID does not exist" Nov 02 14:58:38.110049 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110048 5476 scope.go:122] "RemoveContainer" containerID="7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841" Nov 02 14:58:38.110234 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110219 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841"} err="failed to get container status \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": rpc error: code = NotFound desc = could not find container \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": container with ID starting with 7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841 not found: ID does not exist" Nov 02 14:58:38.110276 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110234 5476 scope.go:122] "RemoveContainer" containerID="27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa" Nov 02 14:58:38.110420 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110404 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa"} err="failed to get container status \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": rpc error: code = NotFound desc = could not find container \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": container with ID starting with 27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa not found: ID does not exist" Nov 02 14:58:38.110420 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110419 5476 scope.go:122] "RemoveContainer" containerID="4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4" Nov 02 14:58:38.110586 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110571 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4"} err="failed to get container status \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": rpc error: code = NotFound desc = could not find container \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": container with ID starting with 4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4 not found: ID does not exist" Nov 02 14:58:38.110629 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110585 5476 scope.go:122] "RemoveContainer" containerID="5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a" Nov 02 14:58:38.110748 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110733 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a"} err="failed to get container status \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": rpc error: code = NotFound desc = could not find container \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": container with ID starting with 5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a not found: ID does not exist" Nov 02 14:58:38.110748 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110747 5476 scope.go:122] "RemoveContainer" containerID="686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497" Nov 02 14:58:38.110930 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110915 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497"} err="failed to get container status \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": rpc error: code = NotFound desc = could not find container \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": container with ID starting with 686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497 not found: ID does not exist" Nov 02 14:58:38.110930 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.110929 5476 scope.go:122] "RemoveContainer" containerID="8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9" Nov 02 14:58:38.111126 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111110 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9"} err="failed to get container status \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": rpc error: code = NotFound desc = could not find container \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": container with ID starting with 8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9 not found: ID does not exist" Nov 02 14:58:38.111126 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111125 5476 scope.go:122] "RemoveContainer" containerID="60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b" Nov 02 14:58:38.111310 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111293 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b"} err="failed to get container status \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": rpc error: code = NotFound desc = could not find container \"60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b\": container with ID starting with 60d63f1da5e4f0cf4427dc470b53fa38b93de09255841df62476577666ac274b not found: ID does not exist" Nov 02 14:58:38.111310 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111310 5476 scope.go:122] "RemoveContainer" containerID="7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841" Nov 02 14:58:38.111482 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111467 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841"} err="failed to get container status \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": rpc error: code = NotFound desc = could not find container \"7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841\": container with ID starting with 7d8ef56fcf7cd84dd3bbe6eb058f7433bdb4bf5a4cf7e1e7bf840b36bc987841 not found: ID does not exist" Nov 02 14:58:38.111482 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111481 5476 scope.go:122] "RemoveContainer" containerID="27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa" Nov 02 14:58:38.111650 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111634 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa"} err="failed to get container status \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": rpc error: code = NotFound desc = could not find container \"27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa\": container with ID starting with 27d43cfd401083e92c01c938b97e57ec63e7b9a75c72f3efd9f6f70c86c049aa not found: ID does not exist" Nov 02 14:58:38.111650 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111649 5476 scope.go:122] "RemoveContainer" containerID="4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4" Nov 02 14:58:38.111824 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111809 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4"} err="failed to get container status \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": rpc error: code = NotFound desc = could not find container \"4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4\": container with ID starting with 4e5ae9eddf3d53560b3cd599af37a9e5591be1987a28db1f095aae60646f06c4 not found: ID does not exist" Nov 02 14:58:38.111824 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111824 5476 scope.go:122] "RemoveContainer" containerID="5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a" Nov 02 14:58:38.111983 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111967 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a"} err="failed to get container status \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": rpc error: code = NotFound desc = could not find container \"5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a\": container with ID starting with 5a99d81ce85416908981b1233177706929722d0630bf3f90f18063361603c03a not found: ID does not exist" Nov 02 14:58:38.111983 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.111981 5476 scope.go:122] "RemoveContainer" containerID="686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497" Nov 02 14:58:38.112165 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.112150 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497"} err="failed to get container status \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": rpc error: code = NotFound desc = could not find container \"686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497\": container with ID starting with 686e4d9bab107612d25853345ecdf6d64c3eb62a3da1b8a43c7ea652c4a99497 not found: ID does not exist" Nov 02 14:58:38.112206 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.112165 5476 scope.go:122] "RemoveContainer" containerID="8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9" Nov 02 14:58:38.112328 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.112313 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9"} err="failed to get container status \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": rpc error: code = NotFound desc = could not find container \"8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9\": container with ID starting with 8daaf13ba316de03617de24480db4b0a12630efd30829173f7c91750676b88d9 not found: ID does not exist" Nov 02 14:58:38.257834 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.257741 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 14:58:38.303887 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.303845 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 14:58:38.304254 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304235 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 14:58:38.304561 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304535 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="config-reloader" Nov 02 14:58:38.304561 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304555 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="config-reloader" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304577 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="init-config-reloader" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304584 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="init-config-reloader" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304600 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy-metric" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304608 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy-metric" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304621 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy-web" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304628 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy-web" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304635 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304643 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304662 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="prom-label-proxy" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304669 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="prom-label-proxy" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304678 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="alertmanager" Nov 02 14:58:38.304700 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304685 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="alertmanager" Nov 02 14:58:38.305086 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304732 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="prom-label-proxy" Nov 02 14:58:38.305086 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304739 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="alertmanager" Nov 02 14:58:38.305086 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304746 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="config-reloader" Nov 02 14:58:38.305086 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304752 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="init-config-reloader" Nov 02 14:58:38.305086 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304758 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy-metric" Nov 02 14:58:38.305086 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304764 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy-web" Nov 02 14:58:38.305086 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.304769 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" containerName="kube-rbac-proxy" Nov 02 14:58:38.309527 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.309509 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.361716 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.361675 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 14:58:38.361926 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.361799 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 14:58:38.361926 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.361815 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 14:58:38.361926 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.361805 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 14:58:38.361926 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.361898 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 14:58:38.362146 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.361944 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 14:58:38.362146 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.361946 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 14:58:38.362146 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.361966 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-89ttf\"" Nov 02 14:58:38.362146 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.362092 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 14:58:38.407710 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.407683 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 14:58:38.429247 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429210 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429247 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429252 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429446 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429273 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvj7p\" (UniqueName: \"kubernetes.io/projected/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-kube-api-access-pvj7p\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429446 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429408 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-config-out\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429446 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429439 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-web-config\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429581 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429460 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429581 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429501 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429581 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429531 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429581 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429553 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429696 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429595 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429696 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429615 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429696 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429674 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.429783 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.429702 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.530644 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.530560 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.530644 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.530596 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.530644 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.530629 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.530877 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.530650 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.530877 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.530797 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvj7p\" (UniqueName: \"kubernetes.io/projected/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-kube-api-access-pvj7p\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.530877 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.530856 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-config-out\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.531060 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.530896 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-web-config\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.531060 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.530932 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.531060 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.530963 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.531060 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.531016 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.531060 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.531019 5476 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 14:58:38.531299 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.531090 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.531299 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.531215 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.531398 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.531293 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.531398 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.531335 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.531637 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.531610 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.531707 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.531684 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.534566 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.534255 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.534566 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.534381 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.534744 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.534680 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-config-volume\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.534744 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.534687 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.534744 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.534693 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.534744 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.534696 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.535042 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.535017 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.535566 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.535546 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-web-config\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.535795 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.535779 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-config-out\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.536958 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.536473 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 14:58:38.589121 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.589086 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvj7p\" (UniqueName: \"kubernetes.io/projected/cc7b9133-0f4a-4e46-bfe6-7282b3d26e90-kube-api-access-pvj7p\") pod \"alertmanager-main-0\" (UID: \"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:38.633085 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:38.633048 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 14:58:39.061745 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:39.061573 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 14:58:39.065900 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:39.065873 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7b9133_0f4a_4e46_bfe6_7282b3d26e90.slice/crio-09ddde7cb2014d92bf8a29f846daa4a7f88cb0a3aa7e1ac0d20951f9905ad288 WatchSource:0}: Error finding container 09ddde7cb2014d92bf8a29f846daa4a7f88cb0a3aa7e1ac0d20951f9905ad288: Status 404 returned error can't find the container with id 09ddde7cb2014d92bf8a29f846daa4a7f88cb0a3aa7e1ac0d20951f9905ad288 Nov 02 14:58:39.919552 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:39.919516 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b78c94-8df4-49a1-b365-bc0b92eae8ba" path="/var/lib/kubelet/pods/c8b78c94-8df4-49a1-b365-bc0b92eae8ba/volumes" Nov 02 14:58:40.063617 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:40.063584 5476 generic.go:356] "Generic (PLEG): container finished" podID="cc7b9133-0f4a-4e46-bfe6-7282b3d26e90" containerID="80d5d7ae4798897f3ec91cd9cda09a9d6c51e5099e5862809d27747d8fcd96a3" exitCode=0 Nov 02 14:58:40.063951 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:40.063668 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90","Type":"ContainerDied","Data":"80d5d7ae4798897f3ec91cd9cda09a9d6c51e5099e5862809d27747d8fcd96a3"} Nov 02 14:58:40.063951 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:40.063729 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90","Type":"ContainerStarted","Data":"09ddde7cb2014d92bf8a29f846daa4a7f88cb0a3aa7e1ac0d20951f9905ad288"} Nov 02 14:58:41.070349 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:41.070313 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90","Type":"ContainerStarted","Data":"37ed795216ed00c59644f518484430211904bb4afc570fddb7e59dc33eeecac1"} Nov 02 14:58:41.070707 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:41.070383 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90","Type":"ContainerStarted","Data":"50e6cb65acd5957f9040e5f1389a2e4804a81e35e13e2b54265d9ee9de3f0c31"} Nov 02 14:58:41.070707 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:41.070409 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90","Type":"ContainerStarted","Data":"9981d04e76441302aa515301f9455f9435795bea8f747349e02f071b45ac16d5"} Nov 02 14:58:41.070707 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:41.070440 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90","Type":"ContainerStarted","Data":"b30389cbe35eed6e03ab42b3799c1ee57755f5d0a3e7c08e69dad6199e02b43c"} Nov 02 14:58:41.070707 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:41.070464 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90","Type":"ContainerStarted","Data":"ab6126934be68c3a2ece2ab95e9a65f691d0d577cc068145bdcdce24668ff79a"} Nov 02 14:58:41.070707 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:41.070494 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cc7b9133-0f4a-4e46-bfe6-7282b3d26e90","Type":"ContainerStarted","Data":"558814eb2700b41aab558439229e0fb52e6ceb89dd713a57558b4f219088be15"} Nov 02 14:58:41.197896 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:41.197838 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.184168344 podStartE2EDuration="3.197824265s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.655921ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:41.194246601 +0000 UTC m=+23.397124662" watchObservedRunningTime="2026-11-02 14:58:41.197824265 +0000 UTC m=+23.400702315" Nov 02 14:58:42.810360 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:42.810320 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:42.811044 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:42.810599 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="prometheus" containerID="cri-o://ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b" gracePeriod=600 Nov 02 14:58:42.811044 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:42.810628 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy" containerID="cri-o://3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0" gracePeriod=600 Nov 02 14:58:42.811044 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:42.810647 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="thanos-sidecar" containerID="cri-o://2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2" gracePeriod=600 Nov 02 14:58:42.811044 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:42.810676 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy-web" containerID="cri-o://5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1" gracePeriod=600 Nov 02 14:58:42.811044 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:42.810752 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="config-reloader" containerID="cri-o://fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182" gracePeriod=600 Nov 02 14:58:42.811044 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:42.810640 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c" gracePeriod=600 Nov 02 14:58:43.080976 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:43.080902 5476 generic.go:356] "Generic (PLEG): container finished" podID="ddab9016-d633-455c-a5e4-477bad67508d" containerID="3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0" exitCode=0 Nov 02 14:58:43.080976 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:43.080923 5476 generic.go:356] "Generic (PLEG): container finished" podID="ddab9016-d633-455c-a5e4-477bad67508d" containerID="5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1" exitCode=0 Nov 02 14:58:43.080976 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:43.080930 5476 generic.go:356] "Generic (PLEG): container finished" podID="ddab9016-d633-455c-a5e4-477bad67508d" containerID="2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2" exitCode=0 Nov 02 14:58:43.080976 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:43.080937 5476 generic.go:356] "Generic (PLEG): container finished" podID="ddab9016-d633-455c-a5e4-477bad67508d" containerID="fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182" exitCode=0 Nov 02 14:58:43.081218 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:43.080979 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerDied","Data":"3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0"} Nov 02 14:58:43.081218 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:43.081058 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerDied","Data":"5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1"} Nov 02 14:58:43.081218 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:43.081084 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerDied","Data":"2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2"} Nov 02 14:58:43.081218 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:43.081109 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerDied","Data":"fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182"} Nov 02 14:58:44.087806 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:44.087775 5476 generic.go:356] "Generic (PLEG): container finished" podID="ddab9016-d633-455c-a5e4-477bad67508d" containerID="e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c" exitCode=0 Nov 02 14:58:44.088189 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:44.087824 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerDied","Data":"e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c"} Nov 02 14:58:46.355877 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:46.355832 5476 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 02 14:58:46.356275 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:46.355892 5476 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 02 14:58:46.356275 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:46.355916 5476 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 02 14:58:46.356275 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:46.355931 5476 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Nov 02 14:58:49.851132 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.851106 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:49.930342 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930312 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-config-out\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930540 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930347 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-1\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930540 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930378 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-grpc-tls\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930540 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930409 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-config\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930540 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930452 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930540 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930485 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-metrics-client-ca\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930540 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930514 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-tls-assets\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930859 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930545 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-thanos-prometheus-http-client-file\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930859 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930587 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-2\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930859 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930620 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-kube-rbac-proxy\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930859 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930651 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-kube-api-access-vsxp4\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-kube-api-access-vsxp4\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930859 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930691 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-0\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930859 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930711 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-1" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:49.930859 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930723 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-db\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.930859 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930835 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-metrics-client-certs\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.931294 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930892 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-web-config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-web-config\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.931294 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930923 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-trusted-ca-bundle\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.931294 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930952 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-tls\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.931294 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930958 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-metrics-client-ca" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:49.931294 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.930985 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-serving-certs-ca-bundle\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.931294 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.931059 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.931294 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.931106 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-kubelet-serving-ca-bundle\") pod \"ddab9016-d633-455c-a5e4-477bad67508d\" (UID: \"ddab9016-d633-455c-a5e4-477bad67508d\") " Nov 02 14:58:49.931675 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.931352 5476 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:49.931675 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.931375 5476 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-metrics-client-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:49.931783 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.931696 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-db" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:49.931783 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.931757 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-kubelet-serving-ca-bundle" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:49.932840 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.932227 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-serving-certs-ca-bundle" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:49.932840 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.932243 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-trusted-ca-bundle" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:49.933019 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.932959 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-2" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:49.934410 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.934349 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-0" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:49.934605 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.934575 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-kube-rbac-proxy-web" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:49.934723 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.934673 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-config" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:49.935200 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.935151 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-thanos-prometheus-http-client-file" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:49.935310 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.935272 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-thanos-sidecar-tls" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:49.935365 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.935300 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-tls-assets" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:49.935365 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.935359 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-config-out" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:49.935622 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.935546 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-kube-api-access-vsxp4" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "kube-api-access-vsxp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:49.935622 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.935601 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-grpc-tls" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:49.936219 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.936188 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-tls" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:49.936350 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.936250 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-kube-rbac-proxy" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:49.936884 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.936864 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-metrics-client-certs" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:49.945767 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:49.945721 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-web-config" pod "ddab9016-d633-455c-a5e4-477bad67508d" (UID: "ddab9016-d633-455c-a5e4-477bad67508d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:50.032734 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032698 5476 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032734 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032729 5476 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032734 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032739 5476 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-config-out\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032748 5476 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-grpc-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032757 5476 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-config\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032766 5476 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032775 5476 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-tls-assets\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032784 5476 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032794 5476 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032802 5476 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-kube-rbac-proxy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032811 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsxp4\" (UniqueName: \"kubernetes.io/projected/ddab9016-d633-455c-a5e4-477bad67508d-kube-api-access-vsxp4\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032819 5476 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032827 5476 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-k8s-db\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032836 5476 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-metrics-client-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032855 5476 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-web-config\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032863 5476 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032871 5476 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddab9016-d633-455c-a5e4-477bad67508d-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.032963 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.032880 5476 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddab9016-d633-455c-a5e4-477bad67508d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.110321 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.110285 5476 generic.go:356] "Generic (PLEG): container finished" podID="ddab9016-d633-455c-a5e4-477bad67508d" containerID="ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b" exitCode=0 Nov 02 14:58:50.110506 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.110330 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerDied","Data":"ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b"} Nov 02 14:58:50.110506 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.110377 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.110506 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.110402 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddab9016-d633-455c-a5e4-477bad67508d","Type":"ContainerDied","Data":"b7d549e975284753735ba54b893e6a085455f508ae6ee948a321e2b5c09e20da"} Nov 02 14:58:50.110506 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.110437 5476 scope.go:122] "RemoveContainer" containerID="e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c" Nov 02 14:58:50.121836 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.119319 5476 scope.go:122] "RemoveContainer" containerID="3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0" Nov 02 14:58:50.135625 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.135603 5476 scope.go:122] "RemoveContainer" containerID="5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1" Nov 02 14:58:50.142789 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.142764 5476 scope.go:122] "RemoveContainer" containerID="2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2" Nov 02 14:58:50.151633 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.151614 5476 scope.go:122] "RemoveContainer" containerID="fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182" Nov 02 14:58:50.158565 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:50.158536 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 14:58:50.159782 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.159765 5476 scope.go:122] "RemoveContainer" containerID="ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b" Nov 02 14:58:50.167333 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.167310 5476 scope.go:122] "RemoveContainer" containerID="72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe" Nov 02 14:58:50.174666 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.174644 5476 scope.go:122] "RemoveContainer" containerID="e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c" Nov 02 14:58:50.174933 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:50.174907 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c\": container with ID starting with e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c not found: ID does not exist" containerID="e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c" Nov 02 14:58:50.175030 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.174939 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c"} err="failed to get container status \"e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c\": rpc error: code = NotFound desc = could not find container \"e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c\": container with ID starting with e358de4d10a39f2398298bed76716eeb01f50629eecc75a195e7dddea7ebb02c not found: ID does not exist" Nov 02 14:58:50.175030 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.174958 5476 scope.go:122] "RemoveContainer" containerID="3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0" Nov 02 14:58:50.175237 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:50.175221 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0\": container with ID starting with 3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0 not found: ID does not exist" containerID="3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0" Nov 02 14:58:50.175303 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.175241 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0"} err="failed to get container status \"3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0\": rpc error: code = NotFound desc = could not find container \"3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0\": container with ID starting with 3f0a06beabc4db84f15af5b98365294f3bde8fe2d32c7d79db4d83908d2372f0 not found: ID does not exist" Nov 02 14:58:50.175303 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.175257 5476 scope.go:122] "RemoveContainer" containerID="5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1" Nov 02 14:58:50.175486 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:50.175466 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1\": container with ID starting with 5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1 not found: ID does not exist" containerID="5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1" Nov 02 14:58:50.175546 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.175487 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1"} err="failed to get container status \"5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1\": rpc error: code = NotFound desc = could not find container \"5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1\": container with ID starting with 5e2acca8de43b62194c4d88a0b37137a6389920c8210824eb15bcce46fa59ce1 not found: ID does not exist" Nov 02 14:58:50.175546 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.175500 5476 scope.go:122] "RemoveContainer" containerID="2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2" Nov 02 14:58:50.175749 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:50.175730 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2\": container with ID starting with 2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2 not found: ID does not exist" containerID="2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2" Nov 02 14:58:50.175793 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.175754 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2"} err="failed to get container status \"2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2\": rpc error: code = NotFound desc = could not find container \"2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2\": container with ID starting with 2a4f2ea909f4d6e4c4207d57d109c7fc998ffd46aaf8f4f7eb4dc34a461415a2 not found: ID does not exist" Nov 02 14:58:50.175793 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.175769 5476 scope.go:122] "RemoveContainer" containerID="fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182" Nov 02 14:58:50.176004 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:50.175977 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182\": container with ID starting with fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182 not found: ID does not exist" containerID="fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182" Nov 02 14:58:50.176059 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.176017 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182"} err="failed to get container status \"fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182\": rpc error: code = NotFound desc = could not find container \"fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182\": container with ID starting with fbc312aaba47856984742713fa4e1e88ec3661427fb7947d7e87a7823a6f8182 not found: ID does not exist" Nov 02 14:58:50.176059 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.176032 5476 scope.go:122] "RemoveContainer" containerID="ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b" Nov 02 14:58:50.176273 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:50.176253 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b\": container with ID starting with ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b not found: ID does not exist" containerID="ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b" Nov 02 14:58:50.176341 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.176278 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b"} err="failed to get container status \"ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b\": rpc error: code = NotFound desc = could not find container \"ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b\": container with ID starting with ee5f2ff0a0bc9ace3c8aa5fc26540ca6e97818c5d3bf401841b0d8af5a64db8b not found: ID does not exist" Nov 02 14:58:50.176341 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.176298 5476 scope.go:122] "RemoveContainer" containerID="72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe" Nov 02 14:58:50.176518 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:58:50.176502 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe\": container with ID starting with 72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe not found: ID does not exist" containerID="72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe" Nov 02 14:58:50.176575 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.176521 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe"} err="failed to get container status \"72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe\": rpc error: code = NotFound desc = could not find container \"72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe\": container with ID starting with 72cc96e4008366e4c373710de3fe546f0ac23e4bb3b9771db1aff2bce408a0fe not found: ID does not exist" Nov 02 14:58:50.276147 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.276065 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:50.282986 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.282958 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:50.321822 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.321796 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322103 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy-thanos" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322121 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy-thanos" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322138 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="thanos-sidecar" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322147 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="thanos-sidecar" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322157 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="init-config-reloader" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322163 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="init-config-reloader" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322186 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="config-reloader" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322193 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="config-reloader" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322206 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy-web" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322213 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy-web" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322221 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322227 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322244 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="prometheus" Nov 02 14:58:50.322259 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322251 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="prometheus" Nov 02 14:58:50.325153 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322303 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="prometheus" Nov 02 14:58:50.325153 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322313 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy-thanos" Nov 02 14:58:50.325153 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322319 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="thanos-sidecar" Nov 02 14:58:50.325153 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322330 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="config-reloader" Nov 02 14:58:50.325153 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322339 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy-web" Nov 02 14:58:50.325153 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.322347 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddab9016-d633-455c-a5e4-477bad67508d" containerName="kube-rbac-proxy" Nov 02 14:58:50.327821 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.327805 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.382383 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.382357 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 14:58:50.382383 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.382374 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 14:58:50.382602 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.382417 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 14:58:50.382602 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.382453 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 14:58:50.382602 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.382508 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 14:58:50.382602 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.382532 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 14:58:50.382602 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.382540 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 14:58:50.382808 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.382607 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-89xs6\"" Nov 02 14:58:50.382808 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.382647 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 14:58:50.382808 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.382693 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 14:58:50.382808 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.382754 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cp3c23k1g144d\"" Nov 02 14:58:50.383080 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.383065 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 14:58:50.383172 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.383156 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 14:58:50.383326 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.383312 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 14:58:50.383427 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.383413 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 14:58:50.430025 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.429984 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 14:58:50.430187 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.430149 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 14:58:50.430848 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.430824 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:50.436291 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436270 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436383 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436306 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436383 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436338 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436456 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436417 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436456 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436450 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436529 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436473 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436529 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436490 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436627 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436599 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436684 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436628 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436684 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436658 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8vzb\" (UniqueName: \"kubernetes.io/projected/922b435d-9e36-4f79-b74d-c177443df3f0-kube-api-access-z8vzb\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436782 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436738 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/922b435d-9e36-4f79-b74d-c177443df3f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436782 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436766 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436876 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436810 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/922b435d-9e36-4f79-b74d-c177443df3f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436918 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436871 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-config\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436960 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436937 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.436960 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436954 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.437052 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.436978 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.437052 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.437016 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.437131 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.437056 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.437131 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.437081 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538032 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.537913 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/922b435d-9e36-4f79-b74d-c177443df3f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538032 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.537961 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-config\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538032 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.537981 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538032 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538028 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538376 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538059 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538376 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538089 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538376 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538114 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538376 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538139 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538587 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538499 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538587 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538563 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538677 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538597 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538677 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538626 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538677 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538654 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538824 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538686 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538824 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538711 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538824 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538744 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538824 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538770 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.538824 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538799 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vzb\" (UniqueName: \"kubernetes.io/projected/922b435d-9e36-4f79-b74d-c177443df3f0-kube-api-access-z8vzb\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.539165 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538836 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/922b435d-9e36-4f79-b74d-c177443df3f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.539165 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.538861 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.540984 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.539606 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.540984 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.540142 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.540984 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.540775 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.541589 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.541431 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.541710 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.541688 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.542020 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.541977 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-config\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.542020 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.541988 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.542256 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.542183 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.542362 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.542342 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/922b435d-9e36-4f79-b74d-c177443df3f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.542793 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.542766 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.542915 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.542806 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.543011 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.542927 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.543408 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.543386 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.543493 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.543403 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.543493 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.543442 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.543585 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.543563 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/922b435d-9e36-4f79-b74d-c177443df3f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.543832 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.543814 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.544168 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.544149 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/922b435d-9e36-4f79-b74d-c177443df3f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.544430 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.544412 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/922b435d-9e36-4f79-b74d-c177443df3f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.598789 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.598759 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vzb\" (UniqueName: \"kubernetes.io/projected/922b435d-9e36-4f79-b74d-c177443df3f0-kube-api-access-z8vzb\") pod \"prometheus-k8s-0\" (UID: \"922b435d-9e36-4f79-b74d-c177443df3f0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:50.652780 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:50.652750 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:58:51.082832 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:51.082797 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 14:58:51.090678 ip-10-0-142-141 kubenswrapper[5476]: W1102 14:58:51.090650 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922b435d_9e36_4f79_b74d_c177443df3f0.slice/crio-d055809b16e7c2ccef147bace8e4fc2eb819c7f1d8019786ab7a8609f8407d34 WatchSource:0}: Error finding container d055809b16e7c2ccef147bace8e4fc2eb819c7f1d8019786ab7a8609f8407d34: Status 404 returned error can't find the container with id d055809b16e7c2ccef147bace8e4fc2eb819c7f1d8019786ab7a8609f8407d34 Nov 02 14:58:51.116239 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:51.116214 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"922b435d-9e36-4f79-b74d-c177443df3f0","Type":"ContainerStarted","Data":"d055809b16e7c2ccef147bace8e4fc2eb819c7f1d8019786ab7a8609f8407d34"} Nov 02 14:58:51.919471 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:51.919437 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddab9016-d633-455c-a5e4-477bad67508d" path="/var/lib/kubelet/pods/ddab9016-d633-455c-a5e4-477bad67508d/volumes" Nov 02 14:58:52.121512 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:52.121476 5476 generic.go:356] "Generic (PLEG): container finished" podID="922b435d-9e36-4f79-b74d-c177443df3f0" containerID="d6156e21d5dfdcfd65d5adbd0116c5561888a8375cc680da09cfb2d01a590bdc" exitCode=0 Nov 02 14:58:52.121865 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:52.121558 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"922b435d-9e36-4f79-b74d-c177443df3f0","Type":"ContainerDied","Data":"d6156e21d5dfdcfd65d5adbd0116c5561888a8375cc680da09cfb2d01a590bdc"} Nov 02 14:58:52.737663 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:52.737594 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vxjb7" Nov 02 14:58:53.128912 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:53.128877 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"922b435d-9e36-4f79-b74d-c177443df3f0","Type":"ContainerStarted","Data":"df5886664cbbad73389f314790ce010b6f567fa9de5b413d5b3f6f68de0a5452"} Nov 02 14:58:53.129316 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:53.128941 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"922b435d-9e36-4f79-b74d-c177443df3f0","Type":"ContainerStarted","Data":"3ae297ae2cfa058a4779d0596bbde060bd8f9df3f30e450b9835279aea282946"} Nov 02 14:58:53.129316 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:53.128966 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"922b435d-9e36-4f79-b74d-c177443df3f0","Type":"ContainerStarted","Data":"7d502f2efa9a196c47cdf8709a20ec8168c6001b50b9ec560a95231d925eaf50"} Nov 02 14:58:53.129316 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:53.129012 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"922b435d-9e36-4f79-b74d-c177443df3f0","Type":"ContainerStarted","Data":"755fefde5dd10a11bd48d376d7691370631d14f0092e8173714b30fa62a4802d"} Nov 02 14:58:53.129316 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:53.129036 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"922b435d-9e36-4f79-b74d-c177443df3f0","Type":"ContainerStarted","Data":"4348c323880bf6287a27cc3d1048dd5ee4e8f5408681f1fb84f44ffab9910764"} Nov 02 14:58:53.129316 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:53.129058 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"922b435d-9e36-4f79-b74d-c177443df3f0","Type":"ContainerStarted","Data":"f3918bbbcd5df12ec2ed1e00973d796c494007368e95d328472ae3ca4a844338"} Nov 02 14:58:53.254882 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:53.254823 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.243266697 podStartE2EDuration="3.254806939s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.540242ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:53.254226358 +0000 UTC m=+35.457104414" watchObservedRunningTime="2026-11-02 14:58:53.254806939 +0000 UTC m=+35.457684998" Nov 02 14:58:55.653148 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:58:55.653109 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:59:20.145009 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:59:20.144956 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 14:59:32.015053 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:59:32.014988 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:59:32.030948 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:59:32.030917 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:59:32.274069 ip-10-0-142-141 kubenswrapper[5476]: I1102 14:59:32.273934 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 14:59:50.144819 ip-10-0-142-141 kubenswrapper[5476]: E1102 14:59:50.144778 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:00:20.146836 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:00:20.146754 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:00:50.145045 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:00:50.144984 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:01:20.146156 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:01:20.146119 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:01:50.144681 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:01:50.144604 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:02:20.145030 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:02:20.144972 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:02:50.144462 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:02:50.144421 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:03:17.936604 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:03:17.936572 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:03:17.938065 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:03:17.938045 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:03:20.144466 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:03:20.144433 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:03:50.144371 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:03:50.144337 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:04:20.144674 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:04:20.144639 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:04:23.824719 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.824685 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j"] Nov 02 15:04:23.828114 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.828096 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:04:23.891052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.891015 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:04:23.891176 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.891061 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:04:23.891230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.891168 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r42d2\" (UniqueName: \"kubernetes.io/projected/8eecf753-53b0-40df-a4f9-1a5b7f783524-kube-api-access-r42d2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:04:23.922859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.922825 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 15:04:23.922859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.922858 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 15:04:23.923049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.922908 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-zpxjt\"" Nov 02 15:04:23.923598 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.923586 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j"] Nov 02 15:04:23.924028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.924009 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj"] Nov 02 15:04:23.927372 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.927358 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:04:23.978984 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.978966 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-k48w2\"" Nov 02 15:04:23.979634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.979614 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj"] Nov 02 15:04:23.992240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.992219 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:04:23.992326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.992251 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmtds\" (UniqueName: \"kubernetes.io/projected/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-kube-api-access-dmtds\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:04:23.992326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.992270 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:04:23.992394 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.992336 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:04:23.992394 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.992361 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:04:23.992497 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.992392 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r42d2\" (UniqueName: \"kubernetes.io/projected/8eecf753-53b0-40df-a4f9-1a5b7f783524-kube-api-access-r42d2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:04:23.992569 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.992551 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:04:23.992626 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:23.992586 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:04:24.032553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.032528 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq"] Nov 02 15:04:24.035980 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.035957 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:04:24.049036 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.048986 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r42d2\" (UniqueName: \"kubernetes.io/projected/8eecf753-53b0-40df-a4f9-1a5b7f783524-kube-api-access-r42d2\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:04:24.085371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.085323 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-m96j9\"" Nov 02 15:04:24.092788 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.092764 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:04:24.092899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.092801 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:04:24.092899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.092859 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:04:24.093023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.092918 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s6nj\" (UniqueName: \"kubernetes.io/projected/fc0b9c3b-7453-49e0-843b-65f57e1e2093-kube-api-access-8s6nj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:04:24.093126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.093102 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmtds\" (UniqueName: \"kubernetes.io/projected/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-kube-api-access-dmtds\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:04:24.093203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.093181 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:04:24.093260 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.093215 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:04:24.093441 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.093414 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq"] Nov 02 15:04:24.093675 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.093658 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:04:24.143062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.143039 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf"] Nov 02 15:04:24.143158 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.143096 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:04:24.146326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.146307 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:04:24.156694 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.156675 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmtds\" (UniqueName: \"kubernetes.io/projected/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-kube-api-access-dmtds\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:04:24.198043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.197255 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-h4dp6\"" Nov 02 15:04:24.198043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.197386 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:04:24.198043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.197431 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:04:24.198043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.197481 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:04:24.198043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.197524 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s6nj\" (UniqueName: \"kubernetes.io/projected/fc0b9c3b-7453-49e0-843b-65f57e1e2093-kube-api-access-8s6nj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:04:24.198043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.197553 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcd9x\" (UniqueName: \"kubernetes.io/projected/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-kube-api-access-zcd9x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:04:24.198043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.197596 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:04:24.198043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.197875 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:04:24.198392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.198146 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:04:24.206277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.203594 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf"] Nov 02 15:04:24.226812 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.226788 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969"] Nov 02 15:04:24.230292 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.230272 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:04:24.242071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.242054 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:04:24.255787 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.255765 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s6nj\" (UniqueName: \"kubernetes.io/projected/fc0b9c3b-7453-49e0-843b-65f57e1e2093-kube-api-access-8s6nj\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:04:24.286561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.286247 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969"] Nov 02 15:04:24.297374 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.297350 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-zvb9h\"" Nov 02 15:04:24.298687 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.298658 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:04:24.298795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.298702 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:04:24.298795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.298736 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:04:24.298795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.298763 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl4v4\" (UniqueName: \"kubernetes.io/projected/a2ab4741-6a69-4e6c-b337-632705a06de5-kube-api-access-jl4v4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:04:24.298955 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.298803 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:04:24.298955 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.298870 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcd9x\" (UniqueName: \"kubernetes.io/projected/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-kube-api-access-zcd9x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:04:24.299179 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.299156 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:04:24.299472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.299436 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:04:24.350794 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.350743 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:04:24.356062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.356040 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcd9x\" (UniqueName: \"kubernetes.io/projected/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-kube-api-access-zcd9x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:04:24.399431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.399395 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:04:24.399575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.399460 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl4v4\" (UniqueName: \"kubernetes.io/projected/a2ab4741-6a69-4e6c-b337-632705a06de5-kube-api-access-jl4v4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:04:24.399575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.399496 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:04:24.399895 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.399872 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:04:24.399970 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.399921 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:04:24.456694 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.456663 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl4v4\" (UniqueName: \"kubernetes.io/projected/a2ab4741-6a69-4e6c-b337-632705a06de5-kube-api-access-jl4v4\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:04:24.478484 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.478459 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:04:24.525729 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.525699 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j"] Nov 02 15:04:24.536410 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:04:24.536386 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eecf753_53b0_40df_a4f9_1a5b7f783524.slice/crio-d2bb7acb6fdde04536a11325e8da554cefcf77cd3c05fb631d81f07bf6995298 WatchSource:0}: Error finding container d2bb7acb6fdde04536a11325e8da554cefcf77cd3c05fb631d81f07bf6995298: Status 404 returned error can't find the container with id d2bb7acb6fdde04536a11325e8da554cefcf77cd3c05fb631d81f07bf6995298 Nov 02 15:04:24.538501 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.538482 5476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:04:24.545454 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.545429 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:04:24.606216 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.606188 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj"] Nov 02 15:04:24.613262 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:04:24.613237 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20dcc7d1_a2c9_47d4_93c7_c22eddf60d4b.slice/crio-d4fda622afb1c7039e9233830e78ff308afbf59f85988ae8addf980ed064cab1 WatchSource:0}: Error finding container d4fda622afb1c7039e9233830e78ff308afbf59f85988ae8addf980ed064cab1: Status 404 returned error can't find the container with id d4fda622afb1c7039e9233830e78ff308afbf59f85988ae8addf980ed064cab1 Nov 02 15:04:24.719913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.719882 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq"] Nov 02 15:04:24.726602 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:04:24.726579 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0b9c3b_7453_49e0_843b_65f57e1e2093.slice/crio-f0476dad4f863e1775b5de626985ba6d86f88c02c1be051eca72c7cbcd5665b5 WatchSource:0}: Error finding container f0476dad4f863e1775b5de626985ba6d86f88c02c1be051eca72c7cbcd5665b5: Status 404 returned error can't find the container with id f0476dad4f863e1775b5de626985ba6d86f88c02c1be051eca72c7cbcd5665b5 Nov 02 15:04:24.844495 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.844428 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf"] Nov 02 15:04:24.850976 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:04:24.850948 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffa2cf9_3a2e_4abc_9c7e_78363fabf1e7.slice/crio-3af7329f739cb9cab00193acf678ab01cc49185c2bc5dfd30f577394f8530cbb WatchSource:0}: Error finding container 3af7329f739cb9cab00193acf678ab01cc49185c2bc5dfd30f577394f8530cbb: Status 404 returned error can't find the container with id 3af7329f739cb9cab00193acf678ab01cc49185c2bc5dfd30f577394f8530cbb Nov 02 15:04:24.912371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:24.912310 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969"] Nov 02 15:04:24.919052 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:04:24.919031 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ab4741_6a69_4e6c_b337_632705a06de5.slice/crio-5ee8096f6dfcd16f70d533791be5fefd066d7a99373e182934019fccfe86e94f WatchSource:0}: Error finding container 5ee8096f6dfcd16f70d533791be5fefd066d7a99373e182934019fccfe86e94f: Status 404 returned error can't find the container with id 5ee8096f6dfcd16f70d533791be5fefd066d7a99373e182934019fccfe86e94f Nov 02 15:04:25.218202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.218129 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" event={"ID":"fc0b9c3b-7453-49e0-843b-65f57e1e2093","Type":"ContainerStarted","Data":"f0476dad4f863e1775b5de626985ba6d86f88c02c1be051eca72c7cbcd5665b5"} Nov 02 15:04:25.219276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.219248 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" event={"ID":"8eecf753-53b0-40df-a4f9-1a5b7f783524","Type":"ContainerStarted","Data":"d2bb7acb6fdde04536a11325e8da554cefcf77cd3c05fb631d81f07bf6995298"} Nov 02 15:04:25.220311 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.220284 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" event={"ID":"a2ab4741-6a69-4e6c-b337-632705a06de5","Type":"ContainerStarted","Data":"5ee8096f6dfcd16f70d533791be5fefd066d7a99373e182934019fccfe86e94f"} Nov 02 15:04:25.221395 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.221372 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" event={"ID":"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b","Type":"ContainerStarted","Data":"d4fda622afb1c7039e9233830e78ff308afbf59f85988ae8addf980ed064cab1"} Nov 02 15:04:25.222420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.222399 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" event={"ID":"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7","Type":"ContainerStarted","Data":"3af7329f739cb9cab00193acf678ab01cc49185c2bc5dfd30f577394f8530cbb"} Nov 02 15:04:25.867570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.867536 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82"] Nov 02 15:04:25.871341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.871318 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:04:25.915915 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.915886 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:04:25.916056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.915937 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:04:25.916056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.915960 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvjdw\" (UniqueName: \"kubernetes.io/projected/2e852410-47ae-4e52-9781-410ce2468743-kube-api-access-hvjdw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:04:25.920670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.920648 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-9tw55\"" Nov 02 15:04:25.967708 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:25.967626 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82"] Nov 02 15:04:26.017126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:26.017095 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:04:26.017265 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:26.017152 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvjdw\" (UniqueName: \"kubernetes.io/projected/2e852410-47ae-4e52-9781-410ce2468743-kube-api-access-hvjdw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:04:26.017265 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:26.017228 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:04:26.017645 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:26.017622 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:04:26.017880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:26.017861 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:04:26.078467 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:26.078437 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvjdw\" (UniqueName: \"kubernetes.io/projected/2e852410-47ae-4e52-9781-410ce2468743-kube-api-access-hvjdw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:04:26.189457 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:26.189377 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:04:26.591104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:26.591025 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82"] Nov 02 15:04:26.611406 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:04:26.611373 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e852410_47ae_4e52_9781_410ce2468743.slice/crio-25197304090cd9d7f2aabe81f3288a3b6ca7bd28139b09760e374d5af0873bac WatchSource:0}: Error finding container 25197304090cd9d7f2aabe81f3288a3b6ca7bd28139b09760e374d5af0873bac: Status 404 returned error can't find the container with id 25197304090cd9d7f2aabe81f3288a3b6ca7bd28139b09760e374d5af0873bac Nov 02 15:04:27.240316 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:27.240236 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" event={"ID":"2e852410-47ae-4e52-9781-410ce2468743","Type":"ContainerStarted","Data":"25197304090cd9d7f2aabe81f3288a3b6ca7bd28139b09760e374d5af0873bac"} Nov 02 15:04:31.049280 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.047710 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s"] Nov 02 15:04:31.054264 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.053243 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.103102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.103041 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-r4c8x\"" Nov 02 15:04:31.103102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.103089 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 15:04:31.106859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.106826 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 15:04:31.151140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.151055 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s"] Nov 02 15:04:31.165958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.165833 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.165958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.165879 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.166156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.165988 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.166156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.166054 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.166156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.166095 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.166156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.166125 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.166421 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.166158 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.166421 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.166205 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.166421 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.166232 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktmp2\" (UniqueName: \"kubernetes.io/projected/be550f37-1beb-48d2-90cd-a0b7d8860fd4-kube-api-access-ktmp2\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.203415 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.203391 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.266827 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.266882 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.266926 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.266956 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.267006 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.267050 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.267092 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmp2\" (UniqueName: \"kubernetes.io/projected/be550f37-1beb-48d2-90cd-a0b7d8860fd4-kube-api-access-ktmp2\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.267205 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.267235 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.269620 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.269832 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.270050 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.271384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.270722 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.272333 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.272252 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.278102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.278076 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.279258 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.279197 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.326945 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.326853 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be550f37-1beb-48d2-90cd-a0b7d8860fd4-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.330962 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.330754 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktmp2\" (UniqueName: \"kubernetes.io/projected/be550f37-1beb-48d2-90cd-a0b7d8860fd4-kube-api-access-ktmp2\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s\" (UID: \"be550f37-1beb-48d2-90cd-a0b7d8860fd4\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.373721 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.373411 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:31.798838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.798796 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55"] Nov 02 15:04:31.805272 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.805237 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:04:31.837914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.837884 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7"] Nov 02 15:04:31.842358 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.842336 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:04:31.850416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.850360 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s"] Nov 02 15:04:31.874172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.874148 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:04:31.874298 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.874206 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:04:31.874298 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.874279 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk7qg\" (UniqueName: \"kubernetes.io/projected/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-kube-api-access-pk7qg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:04:31.884354 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.884330 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55"] Nov 02 15:04:31.892573 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:04:31.892540 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe550f37_1beb_48d2_90cd_a0b7d8860fd4.slice/crio-d800448c10c21eba15272e0709166369f61979a8bedbadb14ad367c9831014e2 WatchSource:0}: Error finding container d800448c10c21eba15272e0709166369f61979a8bedbadb14ad367c9831014e2: Status 404 returned error can't find the container with id d800448c10c21eba15272e0709166369f61979a8bedbadb14ad367c9831014e2 Nov 02 15:04:31.897846 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.897736 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7"] Nov 02 15:04:31.975300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.975266 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:04:31.975521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.975323 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk7qg\" (UniqueName: \"kubernetes.io/projected/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-kube-api-access-pk7qg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:04:31.975521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.975384 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:04:31.975521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.975408 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:04:31.975521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.975440 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:04:31.975521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.975473 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4n7\" (UniqueName: \"kubernetes.io/projected/ebff34db-b4f9-4590-a356-790b39979f6f-kube-api-access-vj4n7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:04:31.976134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.976112 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:04:31.976324 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:31.976306 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:04:32.035546 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.035517 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk7qg\" (UniqueName: \"kubernetes.io/projected/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-kube-api-access-pk7qg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:04:32.077890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.076647 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4n7\" (UniqueName: \"kubernetes.io/projected/ebff34db-b4f9-4590-a356-790b39979f6f-kube-api-access-vj4n7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:04:32.077890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.076703 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:04:32.077890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.076768 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:04:32.077890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.077258 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:04:32.077890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.077780 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:04:32.129276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.129047 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:04:32.143806 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.143718 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4n7\" (UniqueName: \"kubernetes.io/projected/ebff34db-b4f9-4590-a356-790b39979f6f-kube-api-access-vj4n7\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:04:32.166960 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.166727 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:04:32.286352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.286321 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" event={"ID":"be550f37-1beb-48d2-90cd-a0b7d8860fd4","Type":"ContainerStarted","Data":"d800448c10c21eba15272e0709166369f61979a8bedbadb14ad367c9831014e2"} Nov 02 15:04:32.580180 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.580008 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7"] Nov 02 15:04:32.604790 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:04:32.604306 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebff34db_b4f9_4590_a356_790b39979f6f.slice/crio-2d6772ecd607a0f15a3cf86309fc1dfda6191d9ae39d77093e108df7de556e79 WatchSource:0}: Error finding container 2d6772ecd607a0f15a3cf86309fc1dfda6191d9ae39d77093e108df7de556e79: Status 404 returned error can't find the container with id 2d6772ecd607a0f15a3cf86309fc1dfda6191d9ae39d77093e108df7de556e79 Nov 02 15:04:32.619017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:32.618937 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55"] Nov 02 15:04:32.639855 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:04:32.639822 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4d1aed_5035_40c2_a471_f5c6e37f7c21.slice/crio-6ef5090e63319c24a066a097e28ce8c473dd0f6e38817222e59b575bf29e12d3 WatchSource:0}: Error finding container 6ef5090e63319c24a066a097e28ce8c473dd0f6e38817222e59b575bf29e12d3: Status 404 returned error can't find the container with id 6ef5090e63319c24a066a097e28ce8c473dd0f6e38817222e59b575bf29e12d3 Nov 02 15:04:33.294092 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:33.293986 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" event={"ID":"fd4d1aed-5035-40c2-a471-f5c6e37f7c21","Type":"ContainerStarted","Data":"6ef5090e63319c24a066a097e28ce8c473dd0f6e38817222e59b575bf29e12d3"} Nov 02 15:04:33.295795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:33.295734 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" event={"ID":"ebff34db-b4f9-4590-a356-790b39979f6f","Type":"ContainerStarted","Data":"2d6772ecd607a0f15a3cf86309fc1dfda6191d9ae39d77093e108df7de556e79"} Nov 02 15:04:40.354331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.354293 5476 generic.go:356] "Generic (PLEG): container finished" podID="8eecf753-53b0-40df-a4f9-1a5b7f783524" containerID="3974a3503143aefbf3fe4e32644b23088817bff0bd4e95784e2aa16a12b729fc" exitCode=0 Nov 02 15:04:40.354793 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.354430 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" event={"ID":"8eecf753-53b0-40df-a4f9-1a5b7f783524","Type":"ContainerDied","Data":"3974a3503143aefbf3fe4e32644b23088817bff0bd4e95784e2aa16a12b729fc"} Nov 02 15:04:40.358720 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.358173 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" event={"ID":"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7","Type":"ContainerDied","Data":"85407fec5c50d9fd39a51bef2e0a8640c6eac56a13cd5d7d80e6fa01ff618a3e"} Nov 02 15:04:40.358900 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.357977 5476 generic.go:356] "Generic (PLEG): container finished" podID="2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" containerID="85407fec5c50d9fd39a51bef2e0a8640c6eac56a13cd5d7d80e6fa01ff618a3e" exitCode=0 Nov 02 15:04:40.362017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.361464 5476 generic.go:356] "Generic (PLEG): container finished" podID="fc0b9c3b-7453-49e0-843b-65f57e1e2093" containerID="75dc66412bf22dafaac95e5c569479be71663ad62ff3b3ac243ad41c74421bc9" exitCode=0 Nov 02 15:04:40.362017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.361536 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" event={"ID":"fc0b9c3b-7453-49e0-843b-65f57e1e2093","Type":"ContainerDied","Data":"75dc66412bf22dafaac95e5c569479be71663ad62ff3b3ac243ad41c74421bc9"} Nov 02 15:04:40.366113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.365764 5476 generic.go:356] "Generic (PLEG): container finished" podID="a2ab4741-6a69-4e6c-b337-632705a06de5" containerID="57ea23cac13310f0ef4d8289d5ceb3be7d70300b1639ee3d067c84ded84ff0fd" exitCode=0 Nov 02 15:04:40.366113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.365846 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" event={"ID":"a2ab4741-6a69-4e6c-b337-632705a06de5","Type":"ContainerDied","Data":"57ea23cac13310f0ef4d8289d5ceb3be7d70300b1639ee3d067c84ded84ff0fd"} Nov 02 15:04:40.368960 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.368880 5476 generic.go:356] "Generic (PLEG): container finished" podID="20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" containerID="6380b315d5c68318dad806b57c81cc19979ceb0063fcfa494fea2770a5d93b1b" exitCode=0 Nov 02 15:04:40.368960 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.368919 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" event={"ID":"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b","Type":"ContainerDied","Data":"6380b315d5c68318dad806b57c81cc19979ceb0063fcfa494fea2770a5d93b1b"} Nov 02 15:04:40.970729 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.970694 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:04:40.970905 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.970774 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:04:40.970905 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:40.970820 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:04:41.379041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:41.378990 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" event={"ID":"be550f37-1beb-48d2-90cd-a0b7d8860fd4","Type":"ContainerStarted","Data":"e1c1dcedd34f21bb77273b6840f2d0acb39918e21fcf91aeeda67c525218f895"} Nov 02 15:04:41.496410 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:41.496359 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" podStartSLOduration=1.421690466 podStartE2EDuration="10.496340702s" totalImagesPullingTime="9.074650236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:41.493611669 +0000 UTC m=+383.696489731" watchObservedRunningTime="2026-11-02 15:04:41.496340702 +0000 UTC m=+383.699218823" Nov 02 15:04:42.373874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.373848 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:42.379564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.379543 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:42.389677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.389652 5476 generic.go:356] "Generic (PLEG): container finished" podID="8eecf753-53b0-40df-a4f9-1a5b7f783524" containerID="8d3b0ba66f0dcbf2323bc1d1bbbcbb7510bc6fc991f12c6a2fbdb312da56a62b" exitCode=0 Nov 02 15:04:42.389786 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.389728 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" event={"ID":"8eecf753-53b0-40df-a4f9-1a5b7f783524","Type":"ContainerDied","Data":"8d3b0ba66f0dcbf2323bc1d1bbbcbb7510bc6fc991f12c6a2fbdb312da56a62b"} Nov 02 15:04:42.391283 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.391258 5476 generic.go:356] "Generic (PLEG): container finished" podID="fd4d1aed-5035-40c2-a471-f5c6e37f7c21" containerID="63ec7a36b1d37a8a98a4ada01b9c758de5770109c1692ca1675c5a4b21d5d7b1" exitCode=0 Nov 02 15:04:42.391372 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.391331 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" event={"ID":"fd4d1aed-5035-40c2-a471-f5c6e37f7c21","Type":"ContainerDied","Data":"63ec7a36b1d37a8a98a4ada01b9c758de5770109c1692ca1675c5a4b21d5d7b1"} Nov 02 15:04:42.393110 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.393092 5476 generic.go:356] "Generic (PLEG): container finished" podID="2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" containerID="8044fc4825ad46ea6145049e178617dce41b1b6a5a23d9e150298f875dbec481" exitCode=0 Nov 02 15:04:42.393310 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.393286 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" event={"ID":"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7","Type":"ContainerDied","Data":"8044fc4825ad46ea6145049e178617dce41b1b6a5a23d9e150298f875dbec481"} Nov 02 15:04:42.394749 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.394713 5476 generic.go:356] "Generic (PLEG): container finished" podID="ebff34db-b4f9-4590-a356-790b39979f6f" containerID="cc46ec64c14091f64b511e9b9242fe0206785b53e7ded9b0cf7439328ded69ba" exitCode=0 Nov 02 15:04:42.394801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.394770 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" event={"ID":"ebff34db-b4f9-4590-a356-790b39979f6f","Type":"ContainerDied","Data":"cc46ec64c14091f64b511e9b9242fe0206785b53e7ded9b0cf7439328ded69ba"} Nov 02 15:04:42.396570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.396548 5476 generic.go:356] "Generic (PLEG): container finished" podID="a2ab4741-6a69-4e6c-b337-632705a06de5" containerID="50ad08b0b94e6cc4c3c91104a1cd12518f3a690183170fc69a4e1c1de203ff77" exitCode=0 Nov 02 15:04:42.396659 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.396571 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" event={"ID":"a2ab4741-6a69-4e6c-b337-632705a06de5","Type":"ContainerDied","Data":"50ad08b0b94e6cc4c3c91104a1cd12518f3a690183170fc69a4e1c1de203ff77"} Nov 02 15:04:42.398140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.398085 5476 generic.go:356] "Generic (PLEG): container finished" podID="2e852410-47ae-4e52-9781-410ce2468743" containerID="735a2261eb790e8bd360f9c37ce07461341d37eef5c86eecc886f136f485c50d" exitCode=0 Nov 02 15:04:42.398140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.398108 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" event={"ID":"2e852410-47ae-4e52-9781-410ce2468743","Type":"ContainerDied","Data":"735a2261eb790e8bd360f9c37ce07461341d37eef5c86eecc886f136f485c50d"} Nov 02 15:04:42.398533 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.398516 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:42.399160 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:42.399145 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s" Nov 02 15:04:43.403913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:43.403884 5476 generic.go:356] "Generic (PLEG): container finished" podID="20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" containerID="caec495894eb735f72173df5f32ec3397775cef0fd6923a83bd85d899f076939" exitCode=0 Nov 02 15:04:43.404328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:43.403979 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" event={"ID":"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b","Type":"ContainerDied","Data":"caec495894eb735f72173df5f32ec3397775cef0fd6923a83bd85d899f076939"} Nov 02 15:04:43.406025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:43.405988 5476 generic.go:356] "Generic (PLEG): container finished" podID="fc0b9c3b-7453-49e0-843b-65f57e1e2093" containerID="596ac88eccadbbc2c353f79f57e11b76583c1573b15b64d6a43deea1cb867da4" exitCode=0 Nov 02 15:04:43.406135 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:43.406092 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" event={"ID":"fc0b9c3b-7453-49e0-843b-65f57e1e2093","Type":"ContainerDied","Data":"596ac88eccadbbc2c353f79f57e11b76583c1573b15b64d6a43deea1cb867da4"} Nov 02 15:04:44.413301 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:44.413196 5476 generic.go:356] "Generic (PLEG): container finished" podID="fd4d1aed-5035-40c2-a471-f5c6e37f7c21" containerID="a5f026f3f3d534f3798c6347805cb2336794524c0e96949e4aeb6813abb7564c" exitCode=0 Nov 02 15:04:44.413774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:44.413349 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" event={"ID":"fd4d1aed-5035-40c2-a471-f5c6e37f7c21","Type":"ContainerDied","Data":"a5f026f3f3d534f3798c6347805cb2336794524c0e96949e4aeb6813abb7564c"} Nov 02 15:04:44.420245 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:44.420220 5476 generic.go:356] "Generic (PLEG): container finished" podID="ebff34db-b4f9-4590-a356-790b39979f6f" containerID="2e52eda60d86b44ea8c063b7cdb5c3c0393e25cd45e3975d6e82443b9afdd5e7" exitCode=0 Nov 02 15:04:44.421087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:44.421065 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" event={"ID":"ebff34db-b4f9-4590-a356-790b39979f6f","Type":"ContainerDied","Data":"2e52eda60d86b44ea8c063b7cdb5c3c0393e25cd45e3975d6e82443b9afdd5e7"} Nov 02 15:04:50.175745 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:04:50.175699 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:04:50.485821 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:50.484709 5476 generic.go:356] "Generic (PLEG): container finished" podID="2e852410-47ae-4e52-9781-410ce2468743" containerID="40295e3c34104f0586ed73b989815a1c7bedfefb395637c92979706d054b7ebd" exitCode=0 Nov 02 15:04:50.485821 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:50.484835 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" event={"ID":"2e852410-47ae-4e52-9781-410ce2468743","Type":"ContainerDied","Data":"40295e3c34104f0586ed73b989815a1c7bedfefb395637c92979706d054b7ebd"} Nov 02 15:04:58.554846 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:58.554118 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" event={"ID":"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7","Type":"ContainerStarted","Data":"98f34ab6de35d088040eaa6303aeed3e82f6ddfcde29687185507251208ba181"} Nov 02 15:04:58.681936 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:58.681445 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" podStartSLOduration=1.934632994 podStartE2EDuration="34.681426617s" totalImagesPullingTime="32.746793623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:58.679306984 +0000 UTC m=+400.882185045" watchObservedRunningTime="2026-11-02 15:04:58.681426617 +0000 UTC m=+400.884304679" Nov 02 15:04:59.569598 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:59.569559 5476 generic.go:356] "Generic (PLEG): container finished" podID="8eecf753-53b0-40df-a4f9-1a5b7f783524" containerID="c121eef363b2d85e50214cf76f8f6c99018cae39516a7d06c48f65fef567bf14" exitCode=0 Nov 02 15:04:59.570039 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:59.569701 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" event={"ID":"8eecf753-53b0-40df-a4f9-1a5b7f783524","Type":"ContainerDied","Data":"c121eef363b2d85e50214cf76f8f6c99018cae39516a7d06c48f65fef567bf14"} Nov 02 15:04:59.574357 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:59.574323 5476 generic.go:356] "Generic (PLEG): container finished" podID="2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" containerID="98f34ab6de35d088040eaa6303aeed3e82f6ddfcde29687185507251208ba181" exitCode=0 Nov 02 15:04:59.574532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:59.574419 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" event={"ID":"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7","Type":"ContainerDied","Data":"98f34ab6de35d088040eaa6303aeed3e82f6ddfcde29687185507251208ba181"} Nov 02 15:04:59.578188 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:59.578139 5476 generic.go:356] "Generic (PLEG): container finished" podID="a2ab4741-6a69-4e6c-b337-632705a06de5" containerID="873de084fb56ca998987a57a16f201c1bc32020b7d46909d07a645fce6d41e99" exitCode=0 Nov 02 15:04:59.578188 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:04:59.578185 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" event={"ID":"a2ab4741-6a69-4e6c-b337-632705a06de5","Type":"ContainerDied","Data":"873de084fb56ca998987a57a16f201c1bc32020b7d46909d07a645fce6d41e99"} Nov 02 15:05:00.148546 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.148324 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn"] Nov 02 15:05:00.152699 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.152183 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn" Nov 02 15:05:00.170819 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.170644 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg8xc\" (UniqueName: \"kubernetes.io/projected/17d1fd25-36c8-4269-ad7d-be7f0281ecf6-kube-api-access-zg8xc\") pod \"jobset-operator-674cd59dc5-rzksn\" (UID: \"17d1fd25-36c8-4269-ad7d-be7f0281ecf6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn" Nov 02 15:05:00.170819 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.170710 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d1fd25-36c8-4269-ad7d-be7f0281ecf6-tmp\") pod \"jobset-operator-674cd59dc5-rzksn\" (UID: \"17d1fd25-36c8-4269-ad7d-be7f0281ecf6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn" Nov 02 15:05:00.201618 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.201438 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-s6jgt\"" Nov 02 15:05:00.201618 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.201532 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 15:05:00.202849 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.202794 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 15:05:00.208312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.208210 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn"] Nov 02 15:05:00.272541 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.271899 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg8xc\" (UniqueName: \"kubernetes.io/projected/17d1fd25-36c8-4269-ad7d-be7f0281ecf6-kube-api-access-zg8xc\") pod \"jobset-operator-674cd59dc5-rzksn\" (UID: \"17d1fd25-36c8-4269-ad7d-be7f0281ecf6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn" Nov 02 15:05:00.272541 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.271964 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d1fd25-36c8-4269-ad7d-be7f0281ecf6-tmp\") pod \"jobset-operator-674cd59dc5-rzksn\" (UID: \"17d1fd25-36c8-4269-ad7d-be7f0281ecf6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn" Nov 02 15:05:00.272541 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.272289 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d1fd25-36c8-4269-ad7d-be7f0281ecf6-tmp\") pod \"jobset-operator-674cd59dc5-rzksn\" (UID: \"17d1fd25-36c8-4269-ad7d-be7f0281ecf6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn" Nov 02 15:05:00.330018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.329929 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg8xc\" (UniqueName: \"kubernetes.io/projected/17d1fd25-36c8-4269-ad7d-be7f0281ecf6-kube-api-access-zg8xc\") pod \"jobset-operator-674cd59dc5-rzksn\" (UID: \"17d1fd25-36c8-4269-ad7d-be7f0281ecf6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn" Nov 02 15:05:00.486794 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.486762 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn" Nov 02 15:05:00.587396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.587172 5476 generic.go:356] "Generic (PLEG): container finished" podID="fc0b9c3b-7453-49e0-843b-65f57e1e2093" containerID="01c8209f04d5152030ce9394ebdf56b87cb2e93b527b1429c9e05875b7ba78be" exitCode=0 Nov 02 15:05:00.587396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.587216 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" event={"ID":"fc0b9c3b-7453-49e0-843b-65f57e1e2093","Type":"ContainerDied","Data":"01c8209f04d5152030ce9394ebdf56b87cb2e93b527b1429c9e05875b7ba78be"} Nov 02 15:05:00.589940 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.589891 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" event={"ID":"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b","Type":"ContainerStarted","Data":"18acd0f5b4c7404c14d52de0633bfa2fc84040421d0831b7abb39253d4e004db"} Nov 02 15:05:00.596176 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.594746 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" event={"ID":"2e852410-47ae-4e52-9781-410ce2468743","Type":"ContainerStarted","Data":"8d961842a8145e9e5643ffe6399e692944ee9b32d3658b498323f23fe323503a"} Nov 02 15:05:00.839763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.839667 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" podStartSLOduration=3.972161773 podStartE2EDuration="37.839648775s" totalImagesPullingTime="33.867487002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:00.837614943 +0000 UTC m=+403.040493003" watchObservedRunningTime="2026-11-02 15:05:00.839648775 +0000 UTC m=+403.042526852" Nov 02 15:05:00.849360 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.849335 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:05:00.857424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.857402 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:05:00.859612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.859590 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:05:00.876013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.875968 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn"] Nov 02 15:05:00.876261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.876244 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-util\" (UniqueName: \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-util\") pod \"a2ab4741-6a69-4e6c-b337-632705a06de5\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " Nov 02 15:05:00.876349 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.876288 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-util\" (UniqueName: \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-util\") pod \"8eecf753-53b0-40df-a4f9-1a5b7f783524\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " Nov 02 15:05:00.876407 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.876348 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-bundle\") pod \"a2ab4741-6a69-4e6c-b337-632705a06de5\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " Nov 02 15:05:00.876453 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.876402 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-kube-api-access-zcd9x\" (UniqueName: \"kubernetes.io/projected/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-kube-api-access-zcd9x\") pod \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " Nov 02 15:05:00.876453 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.876439 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ab4741-6a69-4e6c-b337-632705a06de5-kube-api-access-jl4v4\" (UniqueName: \"kubernetes.io/projected/a2ab4741-6a69-4e6c-b337-632705a06de5-kube-api-access-jl4v4\") pod \"a2ab4741-6a69-4e6c-b337-632705a06de5\" (UID: \"a2ab4741-6a69-4e6c-b337-632705a06de5\") " Nov 02 15:05:00.876518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.876498 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-bundle\") pod \"8eecf753-53b0-40df-a4f9-1a5b7f783524\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " Nov 02 15:05:00.876551 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.876525 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eecf753-53b0-40df-a4f9-1a5b7f783524-kube-api-access-r42d2\" (UniqueName: \"kubernetes.io/projected/8eecf753-53b0-40df-a4f9-1a5b7f783524-kube-api-access-r42d2\") pod \"8eecf753-53b0-40df-a4f9-1a5b7f783524\" (UID: \"8eecf753-53b0-40df-a4f9-1a5b7f783524\") " Nov 02 15:05:00.877170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.876756 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-bundle\") pod \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " Nov 02 15:05:00.877170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.876828 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-util\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-util\") pod \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\" (UID: \"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7\") " Nov 02 15:05:00.877324 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.877185 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-bundle" pod "8eecf753-53b0-40df-a4f9-1a5b7f783524" (UID: "8eecf753-53b0-40df-a4f9-1a5b7f783524"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:00.877324 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.877254 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-bundle" pod "a2ab4741-6a69-4e6c-b337-632705a06de5" (UID: "a2ab4741-6a69-4e6c-b337-632705a06de5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:00.878014 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.877972 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-bundle" pod "2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" (UID: "2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:00.879613 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.879578 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ab4741-6a69-4e6c-b337-632705a06de5-kube-api-access-jl4v4" pod "a2ab4741-6a69-4e6c-b337-632705a06de5" (UID: "a2ab4741-6a69-4e6c-b337-632705a06de5"). InnerVolumeSpecName "kube-api-access-jl4v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:00.879711 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.879659 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eecf753-53b0-40df-a4f9-1a5b7f783524-kube-api-access-r42d2" pod "8eecf753-53b0-40df-a4f9-1a5b7f783524" (UID: "8eecf753-53b0-40df-a4f9-1a5b7f783524"). InnerVolumeSpecName "kube-api-access-r42d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:00.881130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.881081 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-kube-api-access-zcd9x" pod "2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" (UID: "2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7"). InnerVolumeSpecName "kube-api-access-zcd9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:00.883728 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:05:00.883705 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17d1fd25_36c8_4269_ad7d_be7f0281ecf6.slice/crio-74866d0acbd15fa4cb79419a69f4d528fc020c6e0e16ea752fe2442f5552f945 WatchSource:0}: Error finding container 74866d0acbd15fa4cb79419a69f4d528fc020c6e0e16ea752fe2442f5552f945: Status 404 returned error can't find the container with id 74866d0acbd15fa4cb79419a69f4d528fc020c6e0e16ea752fe2442f5552f945 Nov 02 15:05:00.884671 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.884648 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-util" pod "8eecf753-53b0-40df-a4f9-1a5b7f783524" (UID: "8eecf753-53b0-40df-a4f9-1a5b7f783524"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:00.884787 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.884696 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-util" pod "a2ab4741-6a69-4e6c-b337-632705a06de5" (UID: "a2ab4741-6a69-4e6c-b337-632705a06de5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:00.884950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.884925 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-util" pod "2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" (UID: "2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:00.957101 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.957055 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" podStartSLOduration=4.037608544 podStartE2EDuration="35.957037377s" totalImagesPullingTime="31.919428833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:00.956793921 +0000 UTC m=+403.159671982" watchObservedRunningTime="2026-11-02 15:05:00.957037377 +0000 UTC m=+403.159915434" Nov 02 15:05:00.978465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.978321 5476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:00.978465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.978350 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r42d2\" (UniqueName: \"kubernetes.io/projected/8eecf753-53b0-40df-a4f9-1a5b7f783524-kube-api-access-r42d2\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:00.978465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.978365 5476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:00.978465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.978375 5476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-util\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:00.978465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.978388 5476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-util\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:00.978465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.978401 5476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8eecf753-53b0-40df-a4f9-1a5b7f783524-util\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:00.978465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.978414 5476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2ab4741-6a69-4e6c-b337-632705a06de5-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:00.978465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.978427 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcd9x\" (UniqueName: \"kubernetes.io/projected/2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7-kube-api-access-zcd9x\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:00.978465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:00.978443 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl4v4\" (UniqueName: \"kubernetes.io/projected/a2ab4741-6a69-4e6c-b337-632705a06de5-kube-api-access-jl4v4\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:01.599817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.599786 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" event={"ID":"8eecf753-53b0-40df-a4f9-1a5b7f783524","Type":"ContainerDied","Data":"d2bb7acb6fdde04536a11325e8da554cefcf77cd3c05fb631d81f07bf6995298"} Nov 02 15:05:01.600191 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.599815 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4chqw7j" Nov 02 15:05:01.600191 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.599863 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2bb7acb6fdde04536a11325e8da554cefcf77cd3c05fb631d81f07bf6995298" Nov 02 15:05:01.601351 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.601325 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn" event={"ID":"17d1fd25-36c8-4269-ad7d-be7f0281ecf6","Type":"ContainerStarted","Data":"74866d0acbd15fa4cb79419a69f4d528fc020c6e0e16ea752fe2442f5552f945"} Nov 02 15:05:01.603953 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.603921 5476 generic.go:356] "Generic (PLEG): container finished" podID="fd4d1aed-5035-40c2-a471-f5c6e37f7c21" containerID="ab5a4bfbb67b7317b53cc1e33f7eafea7ef688d271a05058d38cc93c00af79f8" exitCode=0 Nov 02 15:05:01.604071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.603966 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" event={"ID":"fd4d1aed-5035-40c2-a471-f5c6e37f7c21","Type":"ContainerDied","Data":"ab5a4bfbb67b7317b53cc1e33f7eafea7ef688d271a05058d38cc93c00af79f8"} Nov 02 15:05:01.605982 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.605959 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" event={"ID":"2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7","Type":"ContainerDied","Data":"3af7329f739cb9cab00193acf678ab01cc49185c2bc5dfd30f577394f8530cbb"} Nov 02 15:05:01.606112 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.606026 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3af7329f739cb9cab00193acf678ab01cc49185c2bc5dfd30f577394f8530cbb" Nov 02 15:05:01.606112 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.606024 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mgzdf" Nov 02 15:05:01.608652 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.608625 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" event={"ID":"ebff34db-b4f9-4590-a356-790b39979f6f","Type":"ContainerStarted","Data":"5c304311a561df607256a56ef2205f6a7a30139a2879a25f3fe4b211a3b12666"} Nov 02 15:05:01.610798 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.610773 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" event={"ID":"a2ab4741-6a69-4e6c-b337-632705a06de5","Type":"ContainerDied","Data":"5ee8096f6dfcd16f70d533791be5fefd066d7a99373e182934019fccfe86e94f"} Nov 02 15:05:01.610886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.610796 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ms969" Nov 02 15:05:01.610886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.610809 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ee8096f6dfcd16f70d533791be5fefd066d7a99373e182934019fccfe86e94f" Nov 02 15:05:01.613139 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.613117 5476 generic.go:356] "Generic (PLEG): container finished" podID="20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" containerID="18acd0f5b4c7404c14d52de0633bfa2fc84040421d0831b7abb39253d4e004db" exitCode=0 Nov 02 15:05:01.613244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.613198 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" event={"ID":"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b","Type":"ContainerDied","Data":"18acd0f5b4c7404c14d52de0633bfa2fc84040421d0831b7abb39253d4e004db"} Nov 02 15:05:01.615548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.615524 5476 generic.go:356] "Generic (PLEG): container finished" podID="2e852410-47ae-4e52-9781-410ce2468743" containerID="8d961842a8145e9e5643ffe6399e692944ee9b32d3658b498323f23fe323503a" exitCode=0 Nov 02 15:05:01.615851 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.615827 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" event={"ID":"2e852410-47ae-4e52-9781-410ce2468743","Type":"ContainerDied","Data":"8d961842a8145e9e5643ffe6399e692944ee9b32d3658b498323f23fe323503a"} Nov 02 15:05:01.740830 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.740809 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:05:01.785211 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.785164 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0b9c3b-7453-49e0-843b-65f57e1e2093-kube-api-access-8s6nj\" (UniqueName: \"kubernetes.io/projected/fc0b9c3b-7453-49e0-843b-65f57e1e2093-kube-api-access-8s6nj\") pod \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " Nov 02 15:05:01.785211 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.785199 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-util\" (UniqueName: \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-util\") pod \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " Nov 02 15:05:01.785386 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.785266 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-bundle\") pod \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\" (UID: \"fc0b9c3b-7453-49e0-843b-65f57e1e2093\") " Nov 02 15:05:01.785745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.785713 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-bundle" pod "fc0b9c3b-7453-49e0-843b-65f57e1e2093" (UID: "fc0b9c3b-7453-49e0-843b-65f57e1e2093"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:01.787182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.787167 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0b9c3b-7453-49e0-843b-65f57e1e2093-kube-api-access-8s6nj" pod "fc0b9c3b-7453-49e0-843b-65f57e1e2093" (UID: "fc0b9c3b-7453-49e0-843b-65f57e1e2093"). InnerVolumeSpecName "kube-api-access-8s6nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:01.792138 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.792117 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-util" pod "fc0b9c3b-7453-49e0-843b-65f57e1e2093" (UID: "fc0b9c3b-7453-49e0-843b-65f57e1e2093"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:01.886879 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.886853 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s6nj\" (UniqueName: \"kubernetes.io/projected/fc0b9c3b-7453-49e0-843b-65f57e1e2093-kube-api-access-8s6nj\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:01.886879 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.886878 5476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-util\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:01.887032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.886892 5476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc0b9c3b-7453-49e0-843b-65f57e1e2093-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:01.986249 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:01.986210 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" podStartSLOduration=3.683025453 podStartE2EDuration="30.986198757s" totalImagesPullingTime="27.303173304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:01.98554748 +0000 UTC m=+404.188425541" watchObservedRunningTime="2026-11-02 15:05:01.986198757 +0000 UTC m=+404.189076816" Nov 02 15:05:02.622160 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.622120 5476 generic.go:356] "Generic (PLEG): container finished" podID="ebff34db-b4f9-4590-a356-790b39979f6f" containerID="5c304311a561df607256a56ef2205f6a7a30139a2879a25f3fe4b211a3b12666" exitCode=0 Nov 02 15:05:02.622560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.622203 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" event={"ID":"ebff34db-b4f9-4590-a356-790b39979f6f","Type":"ContainerDied","Data":"5c304311a561df607256a56ef2205f6a7a30139a2879a25f3fe4b211a3b12666"} Nov 02 15:05:02.623951 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.623929 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" Nov 02 15:05:02.624096 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.623964 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c67ssrq" event={"ID":"fc0b9c3b-7453-49e0-843b-65f57e1e2093","Type":"ContainerDied","Data":"f0476dad4f863e1775b5de626985ba6d86f88c02c1be051eca72c7cbcd5665b5"} Nov 02 15:05:02.624096 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.624006 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0476dad4f863e1775b5de626985ba6d86f88c02c1be051eca72c7cbcd5665b5" Nov 02 15:05:02.756614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.756587 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:05:02.793545 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.793516 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-util\" (UniqueName: \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-util\") pod \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " Nov 02 15:05:02.793652 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.793557 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-bundle\") pod \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " Nov 02 15:05:02.793652 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.793596 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-kube-api-access-pk7qg\" (UniqueName: \"kubernetes.io/projected/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-kube-api-access-pk7qg\") pod \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\" (UID: \"fd4d1aed-5035-40c2-a471-f5c6e37f7c21\") " Nov 02 15:05:02.793974 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.793952 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-bundle" pod "fd4d1aed-5035-40c2-a471-f5c6e37f7c21" (UID: "fd4d1aed-5035-40c2-a471-f5c6e37f7c21"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:02.795562 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.795541 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-kube-api-access-pk7qg" pod "fd4d1aed-5035-40c2-a471-f5c6e37f7c21" (UID: "fd4d1aed-5035-40c2-a471-f5c6e37f7c21"). InnerVolumeSpecName "kube-api-access-pk7qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:02.798123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.798103 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-util" pod "fd4d1aed-5035-40c2-a471-f5c6e37f7c21" (UID: "fd4d1aed-5035-40c2-a471-f5c6e37f7c21"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:02.802578 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.802561 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:05:02.804570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.804553 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:05:02.894977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.894930 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-bundle\") pod \"2e852410-47ae-4e52-9781-410ce2468743\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " Nov 02 15:05:02.894977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.894964 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-bundle\") pod \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " Nov 02 15:05:02.895119 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.895046 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e852410-47ae-4e52-9781-410ce2468743-kube-api-access-hvjdw\" (UniqueName: \"kubernetes.io/projected/2e852410-47ae-4e52-9781-410ce2468743-kube-api-access-hvjdw\") pod \"2e852410-47ae-4e52-9781-410ce2468743\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " Nov 02 15:05:02.895119 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.895109 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-util\" (UniqueName: \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-util\") pod \"2e852410-47ae-4e52-9781-410ce2468743\" (UID: \"2e852410-47ae-4e52-9781-410ce2468743\") " Nov 02 15:05:02.895213 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.895138 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-util\" (UniqueName: \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-util\") pod \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " Nov 02 15:05:02.895309 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.895288 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-kube-api-access-dmtds\" (UniqueName: \"kubernetes.io/projected/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-kube-api-access-dmtds\") pod \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\" (UID: \"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b\") " Nov 02 15:05:02.895559 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.895537 5476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-util\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:02.895643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.895562 5476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:02.895643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.895576 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk7qg\" (UniqueName: \"kubernetes.io/projected/fd4d1aed-5035-40c2-a471-f5c6e37f7c21-kube-api-access-pk7qg\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:02.895788 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.895767 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-bundle" pod "20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" (UID: "20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:02.895916 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.895893 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-bundle" pod "2e852410-47ae-4e52-9781-410ce2468743" (UID: "2e852410-47ae-4e52-9781-410ce2468743"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:02.897170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.897144 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e852410-47ae-4e52-9781-410ce2468743-kube-api-access-hvjdw" pod "2e852410-47ae-4e52-9781-410ce2468743" (UID: "2e852410-47ae-4e52-9781-410ce2468743"). InnerVolumeSpecName "kube-api-access-hvjdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:02.897629 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.897610 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-kube-api-access-dmtds" pod "20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" (UID: "20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b"). InnerVolumeSpecName "kube-api-access-dmtds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:02.899797 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.899778 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-util" pod "20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" (UID: "20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:02.901521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.901500 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-util" pod "2e852410-47ae-4e52-9781-410ce2468743" (UID: "2e852410-47ae-4e52-9781-410ce2468743"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:02.997215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.996964 5476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:02.997215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.997023 5476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:02.997215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.997041 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvjdw\" (UniqueName: \"kubernetes.io/projected/2e852410-47ae-4e52-9781-410ce2468743-kube-api-access-hvjdw\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:02.997215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.997056 5476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e852410-47ae-4e52-9781-410ce2468743-util\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:02.997215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.997070 5476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-util\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:02.997215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:02.997084 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmtds\" (UniqueName: \"kubernetes.io/projected/20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b-kube-api-access-dmtds\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:03.631425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.631392 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" Nov 02 15:05:03.631907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.631389 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62gr55" event={"ID":"fd4d1aed-5035-40c2-a471-f5c6e37f7c21","Type":"ContainerDied","Data":"6ef5090e63319c24a066a097e28ce8c473dd0f6e38817222e59b575bf29e12d3"} Nov 02 15:05:03.631907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.631523 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ef5090e63319c24a066a097e28ce8c473dd0f6e38817222e59b575bf29e12d3" Nov 02 15:05:03.634244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.634217 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" event={"ID":"20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b","Type":"ContainerDied","Data":"d4fda622afb1c7039e9233830e78ff308afbf59f85988ae8addf980ed064cab1"} Nov 02 15:05:03.634409 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.634259 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4fda622afb1c7039e9233830e78ff308afbf59f85988ae8addf980ed064cab1" Nov 02 15:05:03.634409 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.634332 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847an2pjj" Nov 02 15:05:03.641756 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.641734 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" Nov 02 15:05:03.641863 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.641772 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cj5q82" event={"ID":"2e852410-47ae-4e52-9781-410ce2468743","Type":"ContainerDied","Data":"25197304090cd9d7f2aabe81f3288a3b6ca7bd28139b09760e374d5af0873bac"} Nov 02 15:05:03.641863 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.641802 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25197304090cd9d7f2aabe81f3288a3b6ca7bd28139b09760e374d5af0873bac" Nov 02 15:05:03.806091 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.806066 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:05:03.905226 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.905161 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-bundle\") pod \"ebff34db-b4f9-4590-a356-790b39979f6f\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " Nov 02 15:05:03.905226 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.905220 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebff34db-b4f9-4590-a356-790b39979f6f-kube-api-access-vj4n7\" (UniqueName: \"kubernetes.io/projected/ebff34db-b4f9-4590-a356-790b39979f6f-kube-api-access-vj4n7\") pod \"ebff34db-b4f9-4590-a356-790b39979f6f\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " Nov 02 15:05:03.905395 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.905310 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-util\" (UniqueName: \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-util\") pod \"ebff34db-b4f9-4590-a356-790b39979f6f\" (UID: \"ebff34db-b4f9-4590-a356-790b39979f6f\") " Nov 02 15:05:03.906194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.906161 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-bundle" pod "ebff34db-b4f9-4590-a356-790b39979f6f" (UID: "ebff34db-b4f9-4590-a356-790b39979f6f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:03.908154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.908128 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebff34db-b4f9-4590-a356-790b39979f6f-kube-api-access-vj4n7" pod "ebff34db-b4f9-4590-a356-790b39979f6f" (UID: "ebff34db-b4f9-4590-a356-790b39979f6f"). InnerVolumeSpecName "kube-api-access-vj4n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:05:03.912583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:03.912559 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-util" pod "ebff34db-b4f9-4590-a356-790b39979f6f" (UID: "ebff34db-b4f9-4590-a356-790b39979f6f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:05:04.006417 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:04.006385 5476 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-util\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:04.006646 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:04.006619 5476 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebff34db-b4f9-4590-a356-790b39979f6f-bundle\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:04.006755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:04.006650 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj4n7\" (UniqueName: \"kubernetes.io/projected/ebff34db-b4f9-4590-a356-790b39979f6f-kube-api-access-vj4n7\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:05:04.649888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:04.649845 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" event={"ID":"ebff34db-b4f9-4590-a356-790b39979f6f","Type":"ContainerDied","Data":"2d6772ecd607a0f15a3cf86309fc1dfda6191d9ae39d77093e108df7de556e79"} Nov 02 15:05:04.649888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:04.649874 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajcmf7" Nov 02 15:05:04.650397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:04.649907 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d6772ecd607a0f15a3cf86309fc1dfda6191d9ae39d77093e108df7de556e79" Nov 02 15:05:05.656785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:05.656750 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn" event={"ID":"17d1fd25-36c8-4269-ad7d-be7f0281ecf6","Type":"ContainerStarted","Data":"c290b0c4850155e643646dbb08daf0a46c96fd6321171616d0a836c27ab34071"} Nov 02 15:05:05.766297 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:05.766242 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rzksn" podStartSLOduration=1.540425766 podStartE2EDuration="5.766227802s" totalImagesPullingTime="4.225802036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:05.765260997 +0000 UTC m=+407.968139057" watchObservedRunningTime="2026-11-02 15:05:05.766227802 +0000 UTC m=+407.969105861" Nov 02 15:05:08.523831 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.523793 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5"] Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524205 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ebff34db-b4f9-4590-a356-790b39979f6f" containerName="util" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524223 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebff34db-b4f9-4590-a356-790b39979f6f" containerName="util" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524238 5476 cpu_manager.go:414] "containerMap: removing container" podUID="20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" containerName="pull" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524246 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" containerName="pull" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524256 5476 cpu_manager.go:414] "containerMap: removing container" podUID="2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" containerName="extract" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524263 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" containerName="extract" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524271 5476 cpu_manager.go:414] "containerMap: removing container" podUID="20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" containerName="extract" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524278 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" containerName="extract" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524287 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fc0b9c3b-7453-49e0-843b-65f57e1e2093" containerName="util" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524294 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0b9c3b-7453-49e0-843b-65f57e1e2093" containerName="util" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524307 5476 cpu_manager.go:414] "containerMap: removing container" podUID="2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" containerName="pull" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524313 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" containerName="pull" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524321 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a2ab4741-6a69-4e6c-b337-632705a06de5" containerName="extract" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524328 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ab4741-6a69-4e6c-b337-632705a06de5" containerName="extract" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524340 5476 cpu_manager.go:414] "containerMap: removing container" podUID="8eecf753-53b0-40df-a4f9-1a5b7f783524" containerName="extract" Nov 02 15:05:08.524345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524347 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eecf753-53b0-40df-a4f9-1a5b7f783524" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524356 5476 cpu_manager.go:414] "containerMap: removing container" podUID="2e852410-47ae-4e52-9781-410ce2468743" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524362 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e852410-47ae-4e52-9781-410ce2468743" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524372 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ebff34db-b4f9-4590-a356-790b39979f6f" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524379 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebff34db-b4f9-4590-a356-790b39979f6f" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524394 5476 cpu_manager.go:414] "containerMap: removing container" podUID="8eecf753-53b0-40df-a4f9-1a5b7f783524" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524401 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eecf753-53b0-40df-a4f9-1a5b7f783524" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524416 5476 cpu_manager.go:414] "containerMap: removing container" podUID="20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524423 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524432 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a2ab4741-6a69-4e6c-b337-632705a06de5" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524438 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ab4741-6a69-4e6c-b337-632705a06de5" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524448 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fd4d1aed-5035-40c2-a471-f5c6e37f7c21" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524454 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4d1aed-5035-40c2-a471-f5c6e37f7c21" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524466 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ebff34db-b4f9-4590-a356-790b39979f6f" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524473 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebff34db-b4f9-4590-a356-790b39979f6f" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524482 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fd4d1aed-5035-40c2-a471-f5c6e37f7c21" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524490 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4d1aed-5035-40c2-a471-f5c6e37f7c21" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524500 5476 cpu_manager.go:414] "containerMap: removing container" podUID="2e852410-47ae-4e52-9781-410ce2468743" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524506 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e852410-47ae-4e52-9781-410ce2468743" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524519 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fd4d1aed-5035-40c2-a471-f5c6e37f7c21" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524525 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd4d1aed-5035-40c2-a471-f5c6e37f7c21" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524545 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a2ab4741-6a69-4e6c-b337-632705a06de5" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524552 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ab4741-6a69-4e6c-b337-632705a06de5" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524563 5476 cpu_manager.go:414] "containerMap: removing container" podUID="2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524570 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524579 5476 cpu_manager.go:414] "containerMap: removing container" podUID="8eecf753-53b0-40df-a4f9-1a5b7f783524" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524586 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eecf753-53b0-40df-a4f9-1a5b7f783524" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524596 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fc0b9c3b-7453-49e0-843b-65f57e1e2093" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524602 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0b9c3b-7453-49e0-843b-65f57e1e2093" containerName="pull" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524619 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fc0b9c3b-7453-49e0-843b-65f57e1e2093" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524626 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0b9c3b-7453-49e0-843b-65f57e1e2093" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524639 5476 cpu_manager.go:414] "containerMap: removing container" podUID="2e852410-47ae-4e52-9781-410ce2468743" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524648 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e852410-47ae-4e52-9781-410ce2468743" containerName="util" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524717 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd4d1aed-5035-40c2-a471-f5c6e37f7c21" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524731 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0b9c3b-7453-49e0-843b-65f57e1e2093" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524741 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ffa2cf9-3a2e-4abc-9c7e-78363fabf1e7" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524749 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="20dcc7d1-a2c9-47d4-93c7-c22eddf60d4b" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524757 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ab4741-6a69-4e6c-b337-632705a06de5" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524767 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eecf753-53b0-40df-a4f9-1a5b7f783524" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524774 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e852410-47ae-4e52-9781-410ce2468743" containerName="extract" Nov 02 15:05:08.525095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.524783 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebff34db-b4f9-4590-a356-790b39979f6f" containerName="extract" Nov 02 15:05:08.527875 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.527854 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.581565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.581539 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 15:05:08.581663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.581572 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bfjvh\"" Nov 02 15:05:08.582214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.582200 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 15:05:08.582248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.582216 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 15:05:08.628088 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.628063 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 15:05:08.628222 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.628090 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 15:05:08.631469 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.631442 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5"] Nov 02 15:05:08.642312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.642290 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76ad3b2a-ede1-496a-898d-5d73821b03b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.642392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.642327 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcv56\" (UniqueName: \"kubernetes.io/projected/76ad3b2a-ede1-496a-898d-5d73821b03b0-kube-api-access-zcv56\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.642392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.642350 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/76ad3b2a-ede1-496a-898d-5d73821b03b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.642472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.642452 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/76ad3b2a-ede1-496a-898d-5d73821b03b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.642506 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.642478 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76ad3b2a-ede1-496a-898d-5d73821b03b0-tmp\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.642506 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.642499 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/76ad3b2a-ede1-496a-898d-5d73821b03b0-manifests\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.743685 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.743661 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76ad3b2a-ede1-496a-898d-5d73821b03b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.743920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.743706 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcv56\" (UniqueName: \"kubernetes.io/projected/76ad3b2a-ede1-496a-898d-5d73821b03b0-kube-api-access-zcv56\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.743920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.743731 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/76ad3b2a-ede1-496a-898d-5d73821b03b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.743920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.743768 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/76ad3b2a-ede1-496a-898d-5d73821b03b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.743920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.743789 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76ad3b2a-ede1-496a-898d-5d73821b03b0-tmp\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.743920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.743811 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/76ad3b2a-ede1-496a-898d-5d73821b03b0-manifests\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.744261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.744242 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/76ad3b2a-ede1-496a-898d-5d73821b03b0-manifests\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.746310 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.746276 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76ad3b2a-ede1-496a-898d-5d73821b03b0-tmp\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.746411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.746321 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/76ad3b2a-ede1-496a-898d-5d73821b03b0-metrics-cert\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.746411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.746352 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76ad3b2a-ede1-496a-898d-5d73821b03b0-webhook-cert\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.746411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.746385 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/76ad3b2a-ede1-496a-898d-5d73821b03b0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.803601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.803539 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcv56\" (UniqueName: \"kubernetes.io/projected/76ad3b2a-ede1-496a-898d-5d73821b03b0-kube-api-access-zcv56\") pod \"opendatahub-operator-controller-manager-64686958df-p45j5\" (UID: \"76ad3b2a-ede1-496a-898d-5d73821b03b0\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:08.842196 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:08.842172 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:09.343356 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:09.343325 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5"] Nov 02 15:05:09.350104 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:05:09.350078 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ad3b2a_ede1_496a_898d_5d73821b03b0.slice/crio-21671af2f899e90e8a669b38e2a745d603a03ec952c63a6a6f33d54c559b1463 WatchSource:0}: Error finding container 21671af2f899e90e8a669b38e2a745d603a03ec952c63a6a6f33d54c559b1463: Status 404 returned error can't find the container with id 21671af2f899e90e8a669b38e2a745d603a03ec952c63a6a6f33d54c559b1463 Nov 02 15:05:09.676062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:09.676027 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" event={"ID":"76ad3b2a-ede1-496a-898d-5d73821b03b0","Type":"ContainerStarted","Data":"21671af2f899e90e8a669b38e2a745d603a03ec952c63a6a6f33d54c559b1463"} Nov 02 15:05:12.411119 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.411083 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5"] Nov 02 15:05:12.414645 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.414623 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5" Nov 02 15:05:12.463748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.463722 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5"] Nov 02 15:05:12.474603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.474579 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7scb\" (UniqueName: \"kubernetes.io/projected/17566c3b-68fe-4285-9ee4-04244fa016d4-kube-api-access-g7scb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5\" (UID: \"17566c3b-68fe-4285-9ee4-04244fa016d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5" Nov 02 15:05:12.474814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.474744 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17566c3b-68fe-4285-9ee4-04244fa016d4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5\" (UID: \"17566c3b-68fe-4285-9ee4-04244fa016d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5" Nov 02 15:05:12.507804 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.507776 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-dl99d\"" Nov 02 15:05:12.507981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.507787 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 15:05:12.507981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.507787 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 15:05:12.575656 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.575622 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7scb\" (UniqueName: \"kubernetes.io/projected/17566c3b-68fe-4285-9ee4-04244fa016d4-kube-api-access-g7scb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5\" (UID: \"17566c3b-68fe-4285-9ee4-04244fa016d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5" Nov 02 15:05:12.575850 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.575723 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17566c3b-68fe-4285-9ee4-04244fa016d4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5\" (UID: \"17566c3b-68fe-4285-9ee4-04244fa016d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5" Nov 02 15:05:12.576155 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.576135 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17566c3b-68fe-4285-9ee4-04244fa016d4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5\" (UID: \"17566c3b-68fe-4285-9ee4-04244fa016d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5" Nov 02 15:05:12.635093 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.635055 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7scb\" (UniqueName: \"kubernetes.io/projected/17566c3b-68fe-4285-9ee4-04244fa016d4-kube-api-access-g7scb\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5\" (UID: \"17566c3b-68fe-4285-9ee4-04244fa016d4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5" Nov 02 15:05:12.728168 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:12.728076 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5" Nov 02 15:05:13.167418 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:13.167389 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5"] Nov 02 15:05:13.183058 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:05:13.183030 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17566c3b_68fe_4285_9ee4_04244fa016d4.slice/crio-877630cc478b93c4c4ebf306875f4911dc5210052139b1cb4a856fe6750f3373 WatchSource:0}: Error finding container 877630cc478b93c4c4ebf306875f4911dc5210052139b1cb4a856fe6750f3373: Status 404 returned error can't find the container with id 877630cc478b93c4c4ebf306875f4911dc5210052139b1cb4a856fe6750f3373 Nov 02 15:05:13.695763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:13.695733 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5" event={"ID":"17566c3b-68fe-4285-9ee4-04244fa016d4","Type":"ContainerStarted","Data":"877630cc478b93c4c4ebf306875f4911dc5210052139b1cb4a856fe6750f3373"} Nov 02 15:05:13.697446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:13.697422 5476 generic.go:356] "Generic (PLEG): container finished" podID="76ad3b2a-ede1-496a-898d-5d73821b03b0" containerID="09a099e659d9be7b224e713dfba824c8641b31886e258061b7f58610b05c6903" exitCode=0 Nov 02 15:05:13.697544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:13.697500 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" event={"ID":"76ad3b2a-ede1-496a-898d-5d73821b03b0","Type":"ContainerDied","Data":"09a099e659d9be7b224e713dfba824c8641b31886e258061b7f58610b05c6903"} Nov 02 15:05:14.703403 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:14.703368 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" event={"ID":"76ad3b2a-ede1-496a-898d-5d73821b03b0","Type":"ContainerStarted","Data":"75ef065b854653b38539b8c3159b1cac4cb19a7d0d7bfba570b562c25eae1adc"} Nov 02 15:05:14.703756 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:14.703499 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:14.953375 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:14.953320 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" podStartSLOduration=1.782531451 podStartE2EDuration="6.953305885s" totalImagesPullingTime="4.703119506s" totalInitContainerRuntime="467.654928ms" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:14.906653423 +0000 UTC m=+417.109531484" watchObservedRunningTime="2026-11-02 15:05:14.953305885 +0000 UTC m=+417.156183944" Nov 02 15:05:16.714212 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:16.714175 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5" event={"ID":"17566c3b-68fe-4285-9ee4-04244fa016d4","Type":"ContainerStarted","Data":"f3a3c0442140414983aedbc27a919205e5bc1ff41957febf88901d819c40fef1"} Nov 02 15:05:16.828179 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:16.828124 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vnpf5" podStartSLOduration=1.762734305 podStartE2EDuration="4.82811012s" totalImagesPullingTime="3.065375815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:16.826861606 +0000 UTC m=+419.029739660" watchObservedRunningTime="2026-11-02 15:05:16.82811012 +0000 UTC m=+419.030988179" Nov 02 15:05:18.789002 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:18.788957 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl"] Nov 02 15:05:18.795489 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:18.795464 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" Nov 02 15:05:18.897139 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:18.897114 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rnn6s\"" Nov 02 15:05:18.897139 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:18.897138 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 15:05:18.897350 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:18.897341 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 15:05:18.898819 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:18.898799 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl"] Nov 02 15:05:18.926853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:18.926830 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6abac5be-31c8-4f53-a556-cf43374c83a9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4zkcl\" (UID: \"6abac5be-31c8-4f53-a556-cf43374c83a9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" Nov 02 15:05:18.926959 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:18.926859 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrhzd\" (UniqueName: \"kubernetes.io/projected/6abac5be-31c8-4f53-a556-cf43374c83a9-kube-api-access-rrhzd\") pod \"cert-manager-webhook-5dbbbf75cc-4zkcl\" (UID: \"6abac5be-31c8-4f53-a556-cf43374c83a9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" Nov 02 15:05:19.028262 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:19.028240 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6abac5be-31c8-4f53-a556-cf43374c83a9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4zkcl\" (UID: \"6abac5be-31c8-4f53-a556-cf43374c83a9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" Nov 02 15:05:19.028365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:19.028267 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrhzd\" (UniqueName: \"kubernetes.io/projected/6abac5be-31c8-4f53-a556-cf43374c83a9-kube-api-access-rrhzd\") pod \"cert-manager-webhook-5dbbbf75cc-4zkcl\" (UID: \"6abac5be-31c8-4f53-a556-cf43374c83a9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" Nov 02 15:05:19.096465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:19.096399 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6abac5be-31c8-4f53-a556-cf43374c83a9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-4zkcl\" (UID: \"6abac5be-31c8-4f53-a556-cf43374c83a9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" Nov 02 15:05:19.096584 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:19.096569 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrhzd\" (UniqueName: \"kubernetes.io/projected/6abac5be-31c8-4f53-a556-cf43374c83a9-kube-api-access-rrhzd\") pod \"cert-manager-webhook-5dbbbf75cc-4zkcl\" (UID: \"6abac5be-31c8-4f53-a556-cf43374c83a9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" Nov 02 15:05:19.106351 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:19.106329 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" Nov 02 15:05:19.476419 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:19.476389 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl"] Nov 02 15:05:19.482970 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:05:19.482941 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6abac5be_31c8_4f53_a556_cf43374c83a9.slice/crio-744900c1c01b9eb435d202f65dac2058c103b9918eef19cb387c9b08c51f58f2 WatchSource:0}: Error finding container 744900c1c01b9eb435d202f65dac2058c103b9918eef19cb387c9b08c51f58f2: Status 404 returned error can't find the container with id 744900c1c01b9eb435d202f65dac2058c103b9918eef19cb387c9b08c51f58f2 Nov 02 15:05:19.732161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:19.732080 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" event={"ID":"6abac5be-31c8-4f53-a556-cf43374c83a9","Type":"ContainerStarted","Data":"744900c1c01b9eb435d202f65dac2058c103b9918eef19cb387c9b08c51f58f2"} Nov 02 15:05:20.145286 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:05:20.145252 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:05:20.668872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:20.668843 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9"] Nov 02 15:05:20.673494 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:20.673473 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9" Nov 02 15:05:20.728442 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:20.728419 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5px6d\"" Nov 02 15:05:20.742588 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:20.742567 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83256c31-22b9-4dac-9438-3eaa6a0f142d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-d2hj9\" (UID: \"83256c31-22b9-4dac-9438-3eaa6a0f142d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9" Nov 02 15:05:20.742695 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:20.742618 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9drt\" (UniqueName: \"kubernetes.io/projected/83256c31-22b9-4dac-9438-3eaa6a0f142d-kube-api-access-k9drt\") pod \"cert-manager-cainjector-5c57bbd4fc-d2hj9\" (UID: \"83256c31-22b9-4dac-9438-3eaa6a0f142d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9" Nov 02 15:05:20.775724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:20.775699 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9"] Nov 02 15:05:20.844145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:20.844115 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83256c31-22b9-4dac-9438-3eaa6a0f142d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-d2hj9\" (UID: \"83256c31-22b9-4dac-9438-3eaa6a0f142d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9" Nov 02 15:05:20.844260 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:20.844191 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9drt\" (UniqueName: \"kubernetes.io/projected/83256c31-22b9-4dac-9438-3eaa6a0f142d-kube-api-access-k9drt\") pod \"cert-manager-cainjector-5c57bbd4fc-d2hj9\" (UID: \"83256c31-22b9-4dac-9438-3eaa6a0f142d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9" Nov 02 15:05:20.901294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:20.901268 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9drt\" (UniqueName: \"kubernetes.io/projected/83256c31-22b9-4dac-9438-3eaa6a0f142d-kube-api-access-k9drt\") pod \"cert-manager-cainjector-5c57bbd4fc-d2hj9\" (UID: \"83256c31-22b9-4dac-9438-3eaa6a0f142d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9" Nov 02 15:05:20.906620 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:20.906593 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/83256c31-22b9-4dac-9438-3eaa6a0f142d-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-d2hj9\" (UID: \"83256c31-22b9-4dac-9438-3eaa6a0f142d\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9" Nov 02 15:05:20.984602 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:20.984538 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9" Nov 02 15:05:21.354276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:21.354244 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9"] Nov 02 15:05:21.361192 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:05:21.361164 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83256c31_22b9_4dac_9438_3eaa6a0f142d.slice/crio-4edb245a3151d43023ff79fa85aff7cb0bcfc52ea204440d21428240221741fb WatchSource:0}: Error finding container 4edb245a3151d43023ff79fa85aff7cb0bcfc52ea204440d21428240221741fb: Status 404 returned error can't find the container with id 4edb245a3151d43023ff79fa85aff7cb0bcfc52ea204440d21428240221741fb Nov 02 15:05:21.743535 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:21.743444 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9" event={"ID":"83256c31-22b9-4dac-9438-3eaa6a0f142d","Type":"ContainerStarted","Data":"4edb245a3151d43023ff79fa85aff7cb0bcfc52ea204440d21428240221741fb"} Nov 02 15:05:23.755827 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:23.755792 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" event={"ID":"6abac5be-31c8-4f53-a556-cf43374c83a9","Type":"ContainerStarted","Data":"693e03d6edb5446fb3c461d64aa94d6030c0a94fedbc4c509ec8ccddaa14282b"} Nov 02 15:05:23.756259 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:23.755869 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" Nov 02 15:05:23.874442 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:23.874342 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" podStartSLOduration=1.743850874 podStartE2EDuration="5.874296913s" totalImagesPullingTime="4.130446039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:23.874089431 +0000 UTC m=+426.076967492" watchObservedRunningTime="2026-11-02 15:05:23.874296913 +0000 UTC m=+426.077174972" Nov 02 15:05:24.762121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:24.762079 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9" event={"ID":"83256c31-22b9-4dac-9438-3eaa6a0f142d","Type":"ContainerStarted","Data":"07ff508a720873821c25db17f05a817efb2b802c261fd0c579d0d844e007e10d"} Nov 02 15:05:24.873542 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:24.873480 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d2hj9" podStartSLOduration=2.188145193 podStartE2EDuration="4.87346272s" totalImagesPullingTime="2.685317527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:24.873192376 +0000 UTC m=+427.076070436" watchObservedRunningTime="2026-11-02 15:05:24.87346272 +0000 UTC m=+427.076340785" Nov 02 15:05:25.710906 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:25.710872 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-p45j5" Nov 02 15:05:29.764683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:29.764648 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" Nov 02 15:05:38.390872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:38.390840 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m"] Nov 02 15:05:38.398511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:38.398490 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m" Nov 02 15:05:38.451625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:38.451589 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 15:05:38.495921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:38.495893 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfftr\" (UniqueName: \"kubernetes.io/projected/73aedbe1-24c7-4641-bf3f-0f6b9c20688a-kube-api-access-xfftr\") pod \"limitador-operator-controller-manager-5778bb4449-cgq8m\" (UID: \"73aedbe1-24c7-4641-bf3f-0f6b9c20688a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m" Nov 02 15:05:38.497690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:38.497670 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zzds8\"" Nov 02 15:05:38.497809 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:38.497791 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 15:05:38.499371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:38.499352 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m"] Nov 02 15:05:38.596725 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:38.596696 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfftr\" (UniqueName: \"kubernetes.io/projected/73aedbe1-24c7-4641-bf3f-0f6b9c20688a-kube-api-access-xfftr\") pod \"limitador-operator-controller-manager-5778bb4449-cgq8m\" (UID: \"73aedbe1-24c7-4641-bf3f-0f6b9c20688a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m" Nov 02 15:05:38.656367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:38.656302 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfftr\" (UniqueName: \"kubernetes.io/projected/73aedbe1-24c7-4641-bf3f-0f6b9c20688a-kube-api-access-xfftr\") pod \"limitador-operator-controller-manager-5778bb4449-cgq8m\" (UID: \"73aedbe1-24c7-4641-bf3f-0f6b9c20688a\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m" Nov 02 15:05:38.711521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:38.711494 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m" Nov 02 15:05:39.080025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:39.079936 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m"] Nov 02 15:05:39.087096 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:05:39.087062 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73aedbe1_24c7_4641_bf3f_0f6b9c20688a.slice/crio-95e634a10e16ca54a1289013ed0161f6aa3e30d0f13195efaf5e6b8581d88404 WatchSource:0}: Error finding container 95e634a10e16ca54a1289013ed0161f6aa3e30d0f13195efaf5e6b8581d88404: Status 404 returned error can't find the container with id 95e634a10e16ca54a1289013ed0161f6aa3e30d0f13195efaf5e6b8581d88404 Nov 02 15:05:39.829296 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:39.829259 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m" event={"ID":"73aedbe1-24c7-4641-bf3f-0f6b9c20688a","Type":"ContainerStarted","Data":"95e634a10e16ca54a1289013ed0161f6aa3e30d0f13195efaf5e6b8581d88404"} Nov 02 15:05:43.849643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:43.849607 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m" event={"ID":"73aedbe1-24c7-4641-bf3f-0f6b9c20688a","Type":"ContainerStarted","Data":"06e84f299766b1656176d821a8bffb9cdb15122a89631191411f9b46adf07922"} Nov 02 15:05:43.850078 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:43.849766 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m" Nov 02 15:05:43.960093 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:43.960044 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m" podStartSLOduration=2.043774468 podStartE2EDuration="5.960030579s" totalImagesPullingTime="3.916256111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:43.958675369 +0000 UTC m=+446.161553430" watchObservedRunningTime="2026-11-02 15:05:43.960030579 +0000 UTC m=+446.162908686" Nov 02 15:05:44.742873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.742844 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-848b57c455-hljd7"] Nov 02 15:05:44.746367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.746347 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" Nov 02 15:05:44.795680 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.795658 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 15:05:44.842794 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.842767 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-848b57c455-hljd7"] Nov 02 15:05:44.850773 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.850752 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/741af07b-5653-4e7c-969c-469b7ada179b-tmp\") pod \"kube-auth-proxy-848b57c455-hljd7\" (UID: \"741af07b-5653-4e7c-969c-469b7ada179b\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" Nov 02 15:05:44.851172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.850797 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtsq\" (UniqueName: \"kubernetes.io/projected/741af07b-5653-4e7c-969c-469b7ada179b-kube-api-access-8xtsq\") pod \"kube-auth-proxy-848b57c455-hljd7\" (UID: \"741af07b-5653-4e7c-969c-469b7ada179b\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" Nov 02 15:05:44.851172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.850903 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/741af07b-5653-4e7c-969c-469b7ada179b-tls-certs\") pod \"kube-auth-proxy-848b57c455-hljd7\" (UID: \"741af07b-5653-4e7c-969c-469b7ada179b\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" Nov 02 15:05:44.900087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.900065 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 15:05:44.900292 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.900273 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-plrs8\"" Nov 02 15:05:44.952269 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.952243 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/741af07b-5653-4e7c-969c-469b7ada179b-tmp\") pod \"kube-auth-proxy-848b57c455-hljd7\" (UID: \"741af07b-5653-4e7c-969c-469b7ada179b\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" Nov 02 15:05:44.952408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.952277 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtsq\" (UniqueName: \"kubernetes.io/projected/741af07b-5653-4e7c-969c-469b7ada179b-kube-api-access-8xtsq\") pod \"kube-auth-proxy-848b57c455-hljd7\" (UID: \"741af07b-5653-4e7c-969c-469b7ada179b\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" Nov 02 15:05:44.952478 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.952404 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/741af07b-5653-4e7c-969c-469b7ada179b-tls-certs\") pod \"kube-auth-proxy-848b57c455-hljd7\" (UID: \"741af07b-5653-4e7c-969c-469b7ada179b\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" Nov 02 15:05:44.954423 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.954398 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/741af07b-5653-4e7c-969c-469b7ada179b-tmp\") pod \"kube-auth-proxy-848b57c455-hljd7\" (UID: \"741af07b-5653-4e7c-969c-469b7ada179b\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" Nov 02 15:05:44.954742 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:44.954721 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/741af07b-5653-4e7c-969c-469b7ada179b-tls-certs\") pod \"kube-auth-proxy-848b57c455-hljd7\" (UID: \"741af07b-5653-4e7c-969c-469b7ada179b\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" Nov 02 15:05:45.012131 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:45.012071 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtsq\" (UniqueName: \"kubernetes.io/projected/741af07b-5653-4e7c-969c-469b7ada179b-kube-api-access-8xtsq\") pod \"kube-auth-proxy-848b57c455-hljd7\" (UID: \"741af07b-5653-4e7c-969c-469b7ada179b\") " pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" Nov 02 15:05:45.058045 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:45.058016 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" Nov 02 15:05:45.430356 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:45.428635 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-848b57c455-hljd7"] Nov 02 15:05:45.436509 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:05:45.436478 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741af07b_5653_4e7c_969c_469b7ada179b.slice/crio-170903bee8b59a5eeb07afa46c0e8aa14c5c5d4f28394ae575dcaaa43e1deae2 WatchSource:0}: Error finding container 170903bee8b59a5eeb07afa46c0e8aa14c5c5d4f28394ae575dcaaa43e1deae2: Status 404 returned error can't find the container with id 170903bee8b59a5eeb07afa46c0e8aa14c5c5d4f28394ae575dcaaa43e1deae2 Nov 02 15:05:45.860843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:45.860796 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" event={"ID":"741af07b-5653-4e7c-969c-469b7ada179b","Type":"ContainerStarted","Data":"170903bee8b59a5eeb07afa46c0e8aa14c5c5d4f28394ae575dcaaa43e1deae2"} Nov 02 15:05:48.879300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:48.879254 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" event={"ID":"741af07b-5653-4e7c-969c-469b7ada179b","Type":"ContainerStarted","Data":"f25e4a1cf23527232496f092faa6e677e08aa4b9dd2c3d04a16713d9d428e73c"} Nov 02 15:05:48.988507 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:48.988417 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-848b57c455-hljd7" podStartSLOduration=1.7204551590000001 podStartE2EDuration="4.988404216s" totalImagesPullingTime="3.267949057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:48.987520291 +0000 UTC m=+451.190398364" watchObservedRunningTime="2026-11-02 15:05:48.988404216 +0000 UTC m=+451.191282269" Nov 02 15:05:50.146533 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:05:50.146492 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:05:54.856689 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:54.856656 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-cgq8m" Nov 02 15:05:57.747248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.747170 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc"] Nov 02 15:05:57.752267 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.752244 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:57.801160 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.801141 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 15:05:57.801273 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.801170 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 15:05:57.850981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.850953 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhpp5\" (UniqueName: \"kubernetes.io/projected/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-kube-api-access-dhpp5\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:57.851121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.850990 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:57.851189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.851148 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:57.851281 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.851260 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:57.940713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.940680 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-rq6m2\"" Nov 02 15:05:57.940713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.940685 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 15:05:57.942237 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.942217 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc"] Nov 02 15:05:57.952246 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.952219 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:57.952344 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.952283 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:57.952413 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.952382 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhpp5\" (UniqueName: \"kubernetes.io/projected/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-kube-api-access-dhpp5\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:57.952609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.952582 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:57.952609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.952601 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:57.952795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.952666 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:57.953418 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:57.953396 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:58.009817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:58.009756 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhpp5\" (UniqueName: \"kubernetes.io/projected/a1b0feb9-f6d8-4593-a167-6f1b32daafe7-kube-api-access-dhpp5\") pod \"kserve-module-controller-manager-7566cf5fc8-h9cpc\" (UID: \"a1b0feb9-f6d8-4593-a167-6f1b32daafe7\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:58.064889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:58.064864 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:05:58.484912 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:58.484880 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc"] Nov 02 15:05:58.490950 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:05:58.490923 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b0feb9_f6d8_4593_a167_6f1b32daafe7.slice/crio-41b3f17bb6acb26943b314c1e0cce8601d3dac7ba80290c6d7b947d3769f6161 WatchSource:0}: Error finding container 41b3f17bb6acb26943b314c1e0cce8601d3dac7ba80290c6d7b947d3769f6161: Status 404 returned error can't find the container with id 41b3f17bb6acb26943b314c1e0cce8601d3dac7ba80290c6d7b947d3769f6161 Nov 02 15:05:58.924873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:05:58.924844 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" event={"ID":"a1b0feb9-f6d8-4593-a167-6f1b32daafe7","Type":"ContainerStarted","Data":"41b3f17bb6acb26943b314c1e0cce8601d3dac7ba80290c6d7b947d3769f6161"} Nov 02 15:06:00.934639 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:00.934603 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" event={"ID":"a1b0feb9-f6d8-4593-a167-6f1b32daafe7","Type":"ContainerStarted","Data":"38d19518c8ea813941c89674e153fbdc168d8ca75f799ca17237c9301d16cfbd"} Nov 02 15:06:00.935130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:00.934708 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:06:01.095776 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:01.095720 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" podStartSLOduration=2.030093977 podStartE2EDuration="4.095706359s" totalImagesPullingTime="2.065612382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:01.094814966 +0000 UTC m=+463.297693025" watchObservedRunningTime="2026-11-02 15:06:01.095706359 +0000 UTC m=+463.298584419" Nov 02 15:06:05.282656 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.282621 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7f99s"] Nov 02 15:06:05.285574 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.285557 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" Nov 02 15:06:05.294461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.294438 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:06:05.297619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.297604 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" Nov 02 15:06:05.313292 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.313263 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2c9v\" (UniqueName: \"kubernetes.io/projected/5ca74cc4-1d74-4943-a2de-d032d29c4e1b-kube-api-access-k2c9v\") pod \"authorino-56c6c9ddd5-7f99s\" (UID: \"5ca74cc4-1d74-4943-a2de-d032d29c4e1b\") " pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" Nov 02 15:06:05.339175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.339147 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5vb2v\"" Nov 02 15:06:05.340632 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.340608 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7f99s"] Nov 02 15:06:05.351913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.351897 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 15:06:05.382121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.382098 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:06:05.413883 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.413862 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/98f8fe94-c8fe-44ee-8a08-1db2c5489d5f-config-file\") pod \"limitador-limitador-7c9b87d574-7dljq\" (UID: \"98f8fe94-c8fe-44ee-8a08-1db2c5489d5f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" Nov 02 15:06:05.413979 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.413889 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk26r\" (UniqueName: \"kubernetes.io/projected/98f8fe94-c8fe-44ee-8a08-1db2c5489d5f-kube-api-access-rk26r\") pod \"limitador-limitador-7c9b87d574-7dljq\" (UID: \"98f8fe94-c8fe-44ee-8a08-1db2c5489d5f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" Nov 02 15:06:05.413979 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.413924 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2c9v\" (UniqueName: \"kubernetes.io/projected/5ca74cc4-1d74-4943-a2de-d032d29c4e1b-kube-api-access-k2c9v\") pod \"authorino-56c6c9ddd5-7f99s\" (UID: \"5ca74cc4-1d74-4943-a2de-d032d29c4e1b\") " pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" Nov 02 15:06:05.434692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.434674 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-68qgd\"" Nov 02 15:06:05.471172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.471148 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2c9v\" (UniqueName: \"kubernetes.io/projected/5ca74cc4-1d74-4943-a2de-d032d29c4e1b-kube-api-access-k2c9v\") pod \"authorino-56c6c9ddd5-7f99s\" (UID: \"5ca74cc4-1d74-4943-a2de-d032d29c4e1b\") " pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" Nov 02 15:06:05.515287 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.515265 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/98f8fe94-c8fe-44ee-8a08-1db2c5489d5f-config-file\") pod \"limitador-limitador-7c9b87d574-7dljq\" (UID: \"98f8fe94-c8fe-44ee-8a08-1db2c5489d5f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" Nov 02 15:06:05.515387 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.515296 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk26r\" (UniqueName: \"kubernetes.io/projected/98f8fe94-c8fe-44ee-8a08-1db2c5489d5f-kube-api-access-rk26r\") pod \"limitador-limitador-7c9b87d574-7dljq\" (UID: \"98f8fe94-c8fe-44ee-8a08-1db2c5489d5f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" Nov 02 15:06:05.515853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.515836 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/98f8fe94-c8fe-44ee-8a08-1db2c5489d5f-config-file\") pod \"limitador-limitador-7c9b87d574-7dljq\" (UID: \"98f8fe94-c8fe-44ee-8a08-1db2c5489d5f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" Nov 02 15:06:05.572509 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.572460 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk26r\" (UniqueName: \"kubernetes.io/projected/98f8fe94-c8fe-44ee-8a08-1db2c5489d5f-kube-api-access-rk26r\") pod \"limitador-limitador-7c9b87d574-7dljq\" (UID: \"98f8fe94-c8fe-44ee-8a08-1db2c5489d5f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" Nov 02 15:06:05.596255 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.596233 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" Nov 02 15:06:05.610515 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.610492 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" Nov 02 15:06:05.618712 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.618690 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:06:05.963876 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:05.963842 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7f99s"] Nov 02 15:06:05.970076 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:06:05.970049 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ca74cc4_1d74_4943_a2de_d032d29c4e1b.slice/crio-e3404bc4f9d8f36c86dfa6784deceb45f9c9e1ca16190567b54133ff9f9a7581 WatchSource:0}: Error finding container e3404bc4f9d8f36c86dfa6784deceb45f9c9e1ca16190567b54133ff9f9a7581: Status 404 returned error can't find the container with id e3404bc4f9d8f36c86dfa6784deceb45f9c9e1ca16190567b54133ff9f9a7581 Nov 02 15:06:06.016595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:06.016572 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:06:06.022243 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:06:06.022219 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f8fe94_c8fe_44ee_8a08_1db2c5489d5f.slice/crio-6f38fbaae7291ff5a734e4ead8ea6c57503f090d6f5a8a619e22b6ce37869174 WatchSource:0}: Error finding container 6f38fbaae7291ff5a734e4ead8ea6c57503f090d6f5a8a619e22b6ce37869174: Status 404 returned error can't find the container with id 6f38fbaae7291ff5a734e4ead8ea6c57503f090d6f5a8a619e22b6ce37869174 Nov 02 15:06:06.960973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:06.960936 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" event={"ID":"98f8fe94-c8fe-44ee-8a08-1db2c5489d5f","Type":"ContainerStarted","Data":"6f38fbaae7291ff5a734e4ead8ea6c57503f090d6f5a8a619e22b6ce37869174"} Nov 02 15:06:06.962072 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:06.962039 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" event={"ID":"5ca74cc4-1d74-4943-a2de-d032d29c4e1b","Type":"ContainerStarted","Data":"e3404bc4f9d8f36c86dfa6784deceb45f9c9e1ca16190567b54133ff9f9a7581"} Nov 02 15:06:08.984275 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:08.984232 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" event={"ID":"98f8fe94-c8fe-44ee-8a08-1db2c5489d5f","Type":"ContainerStarted","Data":"a207566dbdec2b35955241fbfb9f93688e24e46eab93bb6dfbfb10b5af17264f"} Nov 02 15:06:08.984700 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:08.984374 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" Nov 02 15:06:09.102109 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:09.102053 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" podStartSLOduration=1.660251681 podStartE2EDuration="4.102039842s" totalImagesPullingTime="2.441788161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:09.100411509 +0000 UTC m=+471.303289570" watchObservedRunningTime="2026-11-02 15:06:09.102039842 +0000 UTC m=+471.304917893" Nov 02 15:06:09.989750 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:09.989706 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" event={"ID":"5ca74cc4-1d74-4943-a2de-d032d29c4e1b","Type":"ContainerStarted","Data":"145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20"} Nov 02 15:06:10.097204 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:10.097154 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" podStartSLOduration=1.912721559 podStartE2EDuration="5.097139282s" totalImagesPullingTime="3.184417723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:10.09653836 +0000 UTC m=+472.299416422" watchObservedRunningTime="2026-11-02 15:06:10.097139282 +0000 UTC m=+472.300017394" Nov 02 15:06:19.991572 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:19.991545 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-7dljq" Nov 02 15:06:20.152496 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:06:20.147511 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:06:21.939986 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:06:21.939957 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-h9cpc" Nov 02 15:06:50.145816 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:06:50.145780 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:07:02.536874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.536842 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pmdck"] Nov 02 15:07:02.540287 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.540271 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" Nov 02 15:07:02.578365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.578338 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtstl\" (UniqueName: \"kubernetes.io/projected/3726b436-7706-438d-90a2-d804726c1399-kube-api-access-mtstl\") pod \"kserve-controller-manager-776cc66d47-pmdck\" (UID: \"3726b436-7706-438d-90a2-d804726c1399\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" Nov 02 15:07:02.578488 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.578371 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3726b436-7706-438d-90a2-d804726c1399-cert\") pod \"kserve-controller-manager-776cc66d47-pmdck\" (UID: \"3726b436-7706-438d-90a2-d804726c1399\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" Nov 02 15:07:02.635658 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.635627 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 02 15:07:02.635792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.635690 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-mv5sr\"" Nov 02 15:07:02.637297 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.637276 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pmdck"] Nov 02 15:07:02.679571 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.679531 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtstl\" (UniqueName: \"kubernetes.io/projected/3726b436-7706-438d-90a2-d804726c1399-kube-api-access-mtstl\") pod \"kserve-controller-manager-776cc66d47-pmdck\" (UID: \"3726b436-7706-438d-90a2-d804726c1399\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" Nov 02 15:07:02.679571 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.679575 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3726b436-7706-438d-90a2-d804726c1399-cert\") pod \"kserve-controller-manager-776cc66d47-pmdck\" (UID: \"3726b436-7706-438d-90a2-d804726c1399\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" Nov 02 15:07:02.682071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.682046 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3726b436-7706-438d-90a2-d804726c1399-cert\") pod \"kserve-controller-manager-776cc66d47-pmdck\" (UID: \"3726b436-7706-438d-90a2-d804726c1399\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" Nov 02 15:07:02.745323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.745297 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtstl\" (UniqueName: \"kubernetes.io/projected/3726b436-7706-438d-90a2-d804726c1399-kube-api-access-mtstl\") pod \"kserve-controller-manager-776cc66d47-pmdck\" (UID: \"3726b436-7706-438d-90a2-d804726c1399\") " pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" Nov 02 15:07:02.774765 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.774738 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cn7hn"] Nov 02 15:07:02.778419 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.778400 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" Nov 02 15:07:02.779965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.779943 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351c7a71-9451-4c65-9db0-a5d8d5d24c39-tls-certs\") pod \"model-serving-api-75bd89fdfc-cn7hn\" (UID: \"351c7a71-9451-4c65-9db0-a5d8d5d24c39\") " pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" Nov 02 15:07:02.780075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.780031 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n64z\" (UniqueName: \"kubernetes.io/projected/351c7a71-9451-4c65-9db0-a5d8d5d24c39-kube-api-access-7n64z\") pod \"model-serving-api-75bd89fdfc-cn7hn\" (UID: \"351c7a71-9451-4c65-9db0-a5d8d5d24c39\") " pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" Nov 02 15:07:02.827604 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.827553 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-c7c5r\"" Nov 02 15:07:02.827723 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.827694 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 15:07:02.835428 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.835403 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cn7hn"] Nov 02 15:07:02.853049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.853028 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" Nov 02 15:07:02.880561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.880537 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n64z\" (UniqueName: \"kubernetes.io/projected/351c7a71-9451-4c65-9db0-a5d8d5d24c39-kube-api-access-7n64z\") pod \"model-serving-api-75bd89fdfc-cn7hn\" (UID: \"351c7a71-9451-4c65-9db0-a5d8d5d24c39\") " pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" Nov 02 15:07:02.880666 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.880582 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351c7a71-9451-4c65-9db0-a5d8d5d24c39-tls-certs\") pod \"model-serving-api-75bd89fdfc-cn7hn\" (UID: \"351c7a71-9451-4c65-9db0-a5d8d5d24c39\") " pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" Nov 02 15:07:02.882945 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.882922 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/351c7a71-9451-4c65-9db0-a5d8d5d24c39-tls-certs\") pod \"model-serving-api-75bd89fdfc-cn7hn\" (UID: \"351c7a71-9451-4c65-9db0-a5d8d5d24c39\") " pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" Nov 02 15:07:02.937843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:02.937818 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n64z\" (UniqueName: \"kubernetes.io/projected/351c7a71-9451-4c65-9db0-a5d8d5d24c39-kube-api-access-7n64z\") pod \"model-serving-api-75bd89fdfc-cn7hn\" (UID: \"351c7a71-9451-4c65-9db0-a5d8d5d24c39\") " pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" Nov 02 15:07:03.090763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:03.090677 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" Nov 02 15:07:03.225571 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:03.225545 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-pmdck"] Nov 02 15:07:03.231537 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:07:03.231511 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3726b436_7706_438d_90a2_d804726c1399.slice/crio-ac3e32ba7010a2c23c38c614f0300c2a7e2371ae3380b826681d9ef15741e64b WatchSource:0}: Error finding container ac3e32ba7010a2c23c38c614f0300c2a7e2371ae3380b826681d9ef15741e64b: Status 404 returned error can't find the container with id ac3e32ba7010a2c23c38c614f0300c2a7e2371ae3380b826681d9ef15741e64b Nov 02 15:07:03.240475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:03.240451 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" event={"ID":"3726b436-7706-438d-90a2-d804726c1399","Type":"ContainerStarted","Data":"ac3e32ba7010a2c23c38c614f0300c2a7e2371ae3380b826681d9ef15741e64b"} Nov 02 15:07:03.458271 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:03.458245 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cn7hn"] Nov 02 15:07:03.464665 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:07:03.464640 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351c7a71_9451_4c65_9db0_a5d8d5d24c39.slice/crio-6e4686008e5d549114f1c8b152c88d156372f7eb59a663439a322ef35f3fcd75 WatchSource:0}: Error finding container 6e4686008e5d549114f1c8b152c88d156372f7eb59a663439a322ef35f3fcd75: Status 404 returned error can't find the container with id 6e4686008e5d549114f1c8b152c88d156372f7eb59a663439a322ef35f3fcd75 Nov 02 15:07:04.247094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:04.247061 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" event={"ID":"351c7a71-9451-4c65-9db0-a5d8d5d24c39","Type":"ContainerStarted","Data":"6e4686008e5d549114f1c8b152c88d156372f7eb59a663439a322ef35f3fcd75"} Nov 02 15:07:06.264442 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:06.264345 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" event={"ID":"351c7a71-9451-4c65-9db0-a5d8d5d24c39","Type":"ContainerStarted","Data":"c082027e725dee58600c26e156b0676817692784e241b4b2c543adf845c6c765"} Nov 02 15:07:06.264899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:06.264527 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" Nov 02 15:07:06.266010 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:06.265969 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" event={"ID":"3726b436-7706-438d-90a2-d804726c1399","Type":"ContainerStarted","Data":"22706c4c081c9511c68628fa640f87124d52cbe721106db47dda3523c8f892b8"} Nov 02 15:07:06.266133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:06.266121 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" Nov 02 15:07:06.375824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:06.375772 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" podStartSLOduration=1.871928224 podStartE2EDuration="4.37575926s" totalImagesPullingTime="2.503831036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:06.374198192 +0000 UTC m=+528.577076249" watchObservedRunningTime="2026-11-02 15:07:06.37575926 +0000 UTC m=+528.578637316" Nov 02 15:07:06.484011 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:06.483960 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" podStartSLOduration=1.757827474 podStartE2EDuration="4.48394776s" totalImagesPullingTime="2.726120286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:06.482675088 +0000 UTC m=+528.685553148" watchObservedRunningTime="2026-11-02 15:07:06.48394776 +0000 UTC m=+528.686825820" Nov 02 15:07:17.275311 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:17.275285 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-cn7hn" Nov 02 15:07:20.145392 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:07:20.145279 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:07:35.148226 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:35.148149 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:07:37.276656 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:37.276632 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-pmdck" Nov 02 15:07:44.875056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:44.875019 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7f99s"] Nov 02 15:07:44.875672 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:44.875180 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" podUID="5ca74cc4-1d74-4943-a2de-d032d29c4e1b" containerName="authorino" containerID="cri-o://145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20" gracePeriod=30 Nov 02 15:07:45.119754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.119731 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" Nov 02 15:07:45.243298 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.243220 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca74cc4-1d74-4943-a2de-d032d29c4e1b-kube-api-access-k2c9v\" (UniqueName: \"kubernetes.io/projected/5ca74cc4-1d74-4943-a2de-d032d29c4e1b-kube-api-access-k2c9v\") pod \"5ca74cc4-1d74-4943-a2de-d032d29c4e1b\" (UID: \"5ca74cc4-1d74-4943-a2de-d032d29c4e1b\") " Nov 02 15:07:45.245382 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.245348 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca74cc4-1d74-4943-a2de-d032d29c4e1b-kube-api-access-k2c9v" pod "5ca74cc4-1d74-4943-a2de-d032d29c4e1b" (UID: "5ca74cc4-1d74-4943-a2de-d032d29c4e1b"). InnerVolumeSpecName "kube-api-access-k2c9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:07:45.344765 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.344740 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2c9v\" (UniqueName: \"kubernetes.io/projected/5ca74cc4-1d74-4943-a2de-d032d29c4e1b-kube-api-access-k2c9v\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:07:45.445146 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.445113 5476 generic.go:356] "Generic (PLEG): container finished" podID="5ca74cc4-1d74-4943-a2de-d032d29c4e1b" containerID="145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20" exitCode=0 Nov 02 15:07:45.445277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.445155 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" event={"ID":"5ca74cc4-1d74-4943-a2de-d032d29c4e1b","Type":"ContainerDied","Data":"145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20"} Nov 02 15:07:45.445277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.445182 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" event={"ID":"5ca74cc4-1d74-4943-a2de-d032d29c4e1b","Type":"ContainerDied","Data":"e3404bc4f9d8f36c86dfa6784deceb45f9c9e1ca16190567b54133ff9f9a7581"} Nov 02 15:07:45.445277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.445185 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-7f99s" Nov 02 15:07:45.445277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.445198 5476 scope.go:122] "RemoveContainer" containerID="145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20" Nov 02 15:07:45.453835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.453818 5476 scope.go:122] "RemoveContainer" containerID="145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20" Nov 02 15:07:45.454132 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:07:45.454114 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20\": container with ID starting with 145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20 not found: ID does not exist" containerID="145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20" Nov 02 15:07:45.454177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.454142 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20"} err="failed to get container status \"145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20\": rpc error: code = NotFound desc = could not find container \"145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20\": container with ID starting with 145d442576b95f0ffdaa492b45ecb978e0f49370d72e8df0088cd86e27f9fa20 not found: ID does not exist" Nov 02 15:07:45.597334 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.597272 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7f99s"] Nov 02 15:07:45.643545 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.643519 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-7f99s"] Nov 02 15:07:45.919977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:07:45.919946 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca74cc4-1d74-4943-a2de-d032d29c4e1b" path="/var/lib/kubelet/pods/5ca74cc4-1d74-4943-a2de-d032d29c4e1b/volumes" Nov 02 15:07:50.145424 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:07:50.145391 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:08:17.975010 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:08:17.974965 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:08:17.975391 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:08:17.975069 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:08:20.145107 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:08:20.145068 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:08:50.146010 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:08:50.145960 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:09:20.146075 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:09:20.146039 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:09:21.887007 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:21.886971 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:09:31.319459 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.319422 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx"] Nov 02 15:09:31.319907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.319788 5476 cpu_manager.go:414] "containerMap: removing container" podUID="5ca74cc4-1d74-4943-a2de-d032d29c4e1b" containerName="authorino" Nov 02 15:09:31.319907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.319799 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca74cc4-1d74-4943-a2de-d032d29c4e1b" containerName="authorino" Nov 02 15:09:31.319907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.319853 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca74cc4-1d74-4943-a2de-d032d29c4e1b" containerName="authorino" Nov 02 15:09:31.322889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.322870 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.372063 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.372039 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 15:09:31.417906 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.417885 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-49zzg\"" Nov 02 15:09:31.418049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.417923 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 15:09:31.418049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.417947 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 15:09:31.419492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.419472 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx"] Nov 02 15:09:31.479877 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.479848 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.480021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.479898 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.480021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.479972 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9wgm\" (UniqueName: \"kubernetes.io/projected/78786688-ccde-4456-8b9e-9cd8ce62ceba-kube-api-access-n9wgm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.480116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.480020 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.480161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.480112 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.480161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.480144 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78786688-ccde-4456-8b9e-9cd8ce62ceba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.581390 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.581318 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9wgm\" (UniqueName: \"kubernetes.io/projected/78786688-ccde-4456-8b9e-9cd8ce62ceba-kube-api-access-n9wgm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.581390 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.581354 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.581560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.581401 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.581560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.581433 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78786688-ccde-4456-8b9e-9cd8ce62ceba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.581560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.581474 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.581560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.581530 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.581843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.581820 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.581885 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.581860 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.581919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.581908 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.583718 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.583698 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78786688-ccde-4456-8b9e-9cd8ce62ceba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.584082 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.584060 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78786688-ccde-4456-8b9e-9cd8ce62ceba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.638938 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.638908 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9wgm\" (UniqueName: \"kubernetes.io/projected/78786688-ccde-4456-8b9e-9cd8ce62ceba-kube-api-access-n9wgm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx\" (UID: \"78786688-ccde-4456-8b9e-9cd8ce62ceba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:31.935039 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:31.935009 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:32.302623 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:32.302553 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx"] Nov 02 15:09:32.308812 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:09:32.308787 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78786688_ccde_4456_8b9e_9cd8ce62ceba.slice/crio-bc5427ac1cd72c8757276ec0f6f9044292fff7f417cc34f60011918b6ffa97f8 WatchSource:0}: Error finding container bc5427ac1cd72c8757276ec0f6f9044292fff7f417cc34f60011918b6ffa97f8: Status 404 returned error can't find the container with id bc5427ac1cd72c8757276ec0f6f9044292fff7f417cc34f60011918b6ffa97f8 Nov 02 15:09:32.310040 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:32.310023 5476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:09:32.956607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:32.956572 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" event={"ID":"78786688-ccde-4456-8b9e-9cd8ce62ceba","Type":"ContainerStarted","Data":"bc5427ac1cd72c8757276ec0f6f9044292fff7f417cc34f60011918b6ffa97f8"} Nov 02 15:09:33.791194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:33.791157 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:09:33.962945 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:33.962911 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" event={"ID":"78786688-ccde-4456-8b9e-9cd8ce62ceba","Type":"ContainerStarted","Data":"cc6dd765a2c86ddff3fb9c6dfe305726344b25fae0d7cbeb872a2e7bf82e2945"} Nov 02 15:09:34.075670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.075580 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" podStartSLOduration=1.5929044239999999 podStartE2EDuration="3.075566952s" totalImagesPullingTime="1.482662528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:34.074790328 +0000 UTC m=+676.277668387" watchObservedRunningTime="2026-11-02 15:09:34.075566952 +0000 UTC m=+676.278445011" Nov 02 15:09:34.817314 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.817285 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rvfxv"] Nov 02 15:09:34.821049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.821028 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:34.913257 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.913221 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:34.913399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.913261 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-home\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:34.913399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.913355 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:34.913399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.913395 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwhpc\" (UniqueName: \"kubernetes.io/projected/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-kube-api-access-hwhpc\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:34.913552 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.913523 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:34.913604 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.913551 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:34.915758 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.915740 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 15:09:34.917495 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.917478 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rvfxv"] Nov 02 15:09:34.936004 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:34.935970 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:35.014368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.014342 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.014785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.014374 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.014785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.014426 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.014785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.014458 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-home\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.014785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.014522 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.014785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.014580 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwhpc\" (UniqueName: \"kubernetes.io/projected/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-kube-api-access-hwhpc\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.015013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.014829 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.015013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.014878 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-home\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.015013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.014899 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.016630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.016614 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.016919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.016903 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.071912 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.071859 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwhpc\" (UniqueName: \"kubernetes.io/projected/b2f3ed9a-28e9-48ad-8ecb-a03c3b779928-kube-api-access-hwhpc\") pod \"echo-simulator-kserve-64b6f8f96c-rvfxv\" (UID: \"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928\") " pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.132962 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.132937 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:35.501286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.501256 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-rvfxv"] Nov 02 15:09:35.507947 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:09:35.507915 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f3ed9a_28e9_48ad_8ecb_a03c3b779928.slice/crio-df7c0a62dae04a301348f68e6b0317213195c3c32759a449ff955aaa8a0fdba1 WatchSource:0}: Error finding container df7c0a62dae04a301348f68e6b0317213195c3c32759a449ff955aaa8a0fdba1: Status 404 returned error can't find the container with id df7c0a62dae04a301348f68e6b0317213195c3c32759a449ff955aaa8a0fdba1 Nov 02 15:09:35.973215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:35.973180 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" event={"ID":"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928","Type":"ContainerStarted","Data":"df7c0a62dae04a301348f68e6b0317213195c3c32759a449ff955aaa8a0fdba1"} Nov 02 15:09:38.990555 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:38.990520 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" event={"ID":"b2f3ed9a-28e9-48ad-8ecb-a03c3b779928","Type":"ContainerStarted","Data":"dabf71efc2855146c1bd0ae034854c48e37c1beba0b1097d1d2758ed49cf74db"} Nov 02 15:09:39.102940 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:39.102864 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" podStartSLOduration=2.64976758 podStartE2EDuration="5.102845872s" totalImagesPullingTime="2.453078292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:39.102540724 +0000 UTC m=+681.305418783" watchObservedRunningTime="2026-11-02 15:09:39.102845872 +0000 UTC m=+681.305723933" Nov 02 15:09:39.133227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:39.133202 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:41.935614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:41.935586 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:41.948116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:41.948093 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:42.015607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.015585 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx" Nov 02 15:09:42.335154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.335085 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj"] Nov 02 15:09:42.338779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.338760 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.378526 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.378492 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.378636 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.378534 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p748k\" (UniqueName: \"kubernetes.io/projected/4350e014-b37d-459a-a887-79d72ed3ed3c-kube-api-access-p748k\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.378636 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.378561 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.378767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.378679 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.378803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.378771 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.378803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.378796 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4350e014-b37d-459a-a887-79d72ed3ed3c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.387786 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.387760 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 15:09:42.395836 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.395812 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj"] Nov 02 15:09:42.479816 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.479793 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.479927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.479825 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4350e014-b37d-459a-a887-79d72ed3ed3c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.480013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.479960 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.480082 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.480016 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p748k\" (UniqueName: \"kubernetes.io/projected/4350e014-b37d-459a-a887-79d72ed3ed3c-kube-api-access-p748k\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.480082 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.480052 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.480176 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.480096 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.480228 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.480216 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.480281 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.480265 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.480587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.480559 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.482194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.482168 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4350e014-b37d-459a-a887-79d72ed3ed3c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.482469 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.482449 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4350e014-b37d-459a-a887-79d72ed3ed3c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.556907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.556883 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p748k\" (UniqueName: \"kubernetes.io/projected/4350e014-b37d-459a-a887-79d72ed3ed3c-kube-api-access-p748k\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj\" (UID: \"4350e014-b37d-459a-a887-79d72ed3ed3c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:42.650085 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:42.650060 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:43.018341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:43.018266 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj"] Nov 02 15:09:43.023772 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:09:43.023749 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4350e014_b37d_459a_a887_79d72ed3ed3c.slice/crio-ad6e58367d2aa323f6e36eabf36b131bdb8378a9ebdd1c7ad7e3a4fe6e259b0e WatchSource:0}: Error finding container ad6e58367d2aa323f6e36eabf36b131bdb8378a9ebdd1c7ad7e3a4fe6e259b0e: Status 404 returned error can't find the container with id ad6e58367d2aa323f6e36eabf36b131bdb8378a9ebdd1c7ad7e3a4fe6e259b0e Nov 02 15:09:44.015642 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:44.015547 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" event={"ID":"4350e014-b37d-459a-a887-79d72ed3ed3c","Type":"ContainerStarted","Data":"d58d1fc9a94853548544dc872909a090b98008c1f4da844d76590059faaa3086"} Nov 02 15:09:44.015642 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:44.015589 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" event={"ID":"4350e014-b37d-459a-a887-79d72ed3ed3c","Type":"ContainerStarted","Data":"ad6e58367d2aa323f6e36eabf36b131bdb8378a9ebdd1c7ad7e3a4fe6e259b0e"} Nov 02 15:09:44.125461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:44.125417 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podStartSLOduration=1.518553513 podStartE2EDuration="2.125405109s" totalImagesPullingTime="606.851596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:44.125036017 +0000 UTC m=+686.327914077" watchObservedRunningTime="2026-11-02 15:09:44.125405109 +0000 UTC m=+686.328283167" Nov 02 15:09:44.650511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:44.650484 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:45.133595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:45.133564 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:45.145817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:45.145794 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:46.036709 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:46.036683 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-rvfxv" Nov 02 15:09:48.251461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.251433 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx"] Nov 02 15:09:48.254911 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.254895 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.314514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.314489 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x6s8k\"" Nov 02 15:09:48.328866 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.328842 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.328958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.328874 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2p64\" (UniqueName: \"kubernetes.io/projected/a71c6c9c-6820-4ced-a069-a20e4dbe3841-kube-api-access-v2p64\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.329047 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.329028 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.329098 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.329065 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.329098 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.329083 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.329171 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.329100 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.329225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.329212 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.329262 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.329237 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.329313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.329262 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.362304 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.362275 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx"] Nov 02 15:09:48.430231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430201 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430244 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430265 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2p64\" (UniqueName: \"kubernetes.io/projected/a71c6c9c-6820-4ced-a069-a20e4dbe3841-kube-api-access-v2p64\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430313 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430468 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430336 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430468 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430354 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430468 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430383 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430468 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430437 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430472 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430649 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430707 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430800 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.430939 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.430917 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.431039 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.431022 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.432507 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.432491 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.432911 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.432892 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.487756 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.487728 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71c6c9c-6820-4ced-a069-a20e4dbe3841-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.487840 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.487786 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2p64\" (UniqueName: \"kubernetes.io/projected/a71c6c9c-6820-4ced-a069-a20e4dbe3841-kube-api-access-v2p64\") pod \"maas-default-gateway-openshift-default-7f5667f86c-256hx\" (UID: \"a71c6c9c-6820-4ced-a069-a20e4dbe3841\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.567664 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.567610 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:48.990194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.990161 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx"] Nov 02 15:09:48.996439 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:09:48.996415 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71c6c9c_6820_4ced_a069_a20e4dbe3841.slice/crio-91cc0d90566badb9a1c9e4728aa7920addbc6c88b70a9bd0b0712cd1e8b46ff5 WatchSource:0}: Error finding container 91cc0d90566badb9a1c9e4728aa7920addbc6c88b70a9bd0b0712cd1e8b46ff5: Status 404 returned error can't find the container with id 91cc0d90566badb9a1c9e4728aa7920addbc6c88b70a9bd0b0712cd1e8b46ff5 Nov 02 15:09:48.998573 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.998540 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:09:48.998659 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.998605 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:09:48.998659 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:48.998633 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:09:49.042060 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:49.042027 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" event={"ID":"a71c6c9c-6820-4ced-a069-a20e4dbe3841","Type":"ContainerStarted","Data":"91cc0d90566badb9a1c9e4728aa7920addbc6c88b70a9bd0b0712cd1e8b46ff5"} Nov 02 15:09:49.652343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:49.652307 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:09:50.049313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:50.049223 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" event={"ID":"a71c6c9c-6820-4ced-a069-a20e4dbe3841","Type":"ContainerStarted","Data":"a1ca5a3ecd2105cd6fd22771b6b0cd8624ae26d7bc393da693fc79f23209dcd4"} Nov 02 15:09:50.152352 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:09:50.152320 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:09:50.211565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:50.211512 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" podStartSLOduration=2.211497466 podStartE2EDuration="2.211497466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:50.210082828 +0000 UTC m=+692.412960904" watchObservedRunningTime="2026-11-02 15:09:50.211497466 +0000 UTC m=+692.414375525" Nov 02 15:09:50.568232 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:50.568197 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:50.569582 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:50.569559 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-256hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 15:09:50.569645 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:50.569605 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" podUID="a71c6c9c-6820-4ced-a069-a20e4dbe3841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 15:09:51.568266 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:51.568230 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-256hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 15:09:51.568667 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:51.568305 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" podUID="a71c6c9c-6820-4ced-a069-a20e4dbe3841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 15:09:52.568038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:52.567975 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-256hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 15:09:52.568247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:52.568071 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" podUID="a71c6c9c-6820-4ced-a069-a20e4dbe3841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 15:09:52.650422 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:52.650387 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:52.662984 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:52.662954 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:53.100638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:53.100601 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:09:53.568932 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:53.568904 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-256hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 15:09:53.569138 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:53.568951 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" podUID="a71c6c9c-6820-4ced-a069-a20e4dbe3841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 15:09:54.568855 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:54.568820 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-256hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 15:09:54.569291 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:54.568877 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" podUID="a71c6c9c-6820-4ced-a069-a20e4dbe3841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 15:09:55.568440 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:55.568402 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-256hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 15:09:55.568643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:55.568473 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" podUID="a71c6c9c-6820-4ced-a069-a20e4dbe3841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 15:09:57.610240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:57.610191 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-256hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:09:57.610730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:57.610272 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" podUID="a71c6c9c-6820-4ced-a069-a20e4dbe3841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:09:58.616838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:58.616805 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-256hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:09:58.617322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:58.616854 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" podUID="a71c6c9c-6820-4ced-a069-a20e4dbe3841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:09:58.619768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:58.619743 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:58.620057 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:58.620034 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:09:58.620560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:09:58.620542 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-256hx" Nov 02 15:10:10.927216 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:10.927181 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56fb547cfb-zdfhv"] Nov 02 15:10:10.935099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:10.935081 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fb547cfb-zdfhv" Nov 02 15:10:10.986149 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:10.986119 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56fb547cfb-zdfhv"] Nov 02 15:10:11.029890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:11.029862 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5vb2v\"" Nov 02 15:10:11.030084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:11.029963 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 02 15:10:11.052581 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:11.052543 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqbtn\" (UniqueName: \"kubernetes.io/projected/5015c227-7a33-4970-86c9-a1a88274a726-kube-api-access-gqbtn\") pod \"authorino-56fb547cfb-zdfhv\" (UID: \"5015c227-7a33-4970-86c9-a1a88274a726\") " pod="kuadrant-system/authorino-56fb547cfb-zdfhv" Nov 02 15:10:11.052761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:11.052609 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5015c227-7a33-4970-86c9-a1a88274a726-tls-cert\") pod \"authorino-56fb547cfb-zdfhv\" (UID: \"5015c227-7a33-4970-86c9-a1a88274a726\") " pod="kuadrant-system/authorino-56fb547cfb-zdfhv" Nov 02 15:10:11.153693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:11.153661 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqbtn\" (UniqueName: \"kubernetes.io/projected/5015c227-7a33-4970-86c9-a1a88274a726-kube-api-access-gqbtn\") pod \"authorino-56fb547cfb-zdfhv\" (UID: \"5015c227-7a33-4970-86c9-a1a88274a726\") " pod="kuadrant-system/authorino-56fb547cfb-zdfhv" Nov 02 15:10:11.153882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:11.153715 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5015c227-7a33-4970-86c9-a1a88274a726-tls-cert\") pod \"authorino-56fb547cfb-zdfhv\" (UID: \"5015c227-7a33-4970-86c9-a1a88274a726\") " pod="kuadrant-system/authorino-56fb547cfb-zdfhv" Nov 02 15:10:11.156339 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:11.156310 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5015c227-7a33-4970-86c9-a1a88274a726-tls-cert\") pod \"authorino-56fb547cfb-zdfhv\" (UID: \"5015c227-7a33-4970-86c9-a1a88274a726\") " pod="kuadrant-system/authorino-56fb547cfb-zdfhv" Nov 02 15:10:11.210930 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:11.210864 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqbtn\" (UniqueName: \"kubernetes.io/projected/5015c227-7a33-4970-86c9-a1a88274a726-kube-api-access-gqbtn\") pod \"authorino-56fb547cfb-zdfhv\" (UID: \"5015c227-7a33-4970-86c9-a1a88274a726\") " pod="kuadrant-system/authorino-56fb547cfb-zdfhv" Nov 02 15:10:11.247838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:11.247813 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56fb547cfb-zdfhv" Nov 02 15:10:11.617980 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:11.617943 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56fb547cfb-zdfhv"] Nov 02 15:10:11.624030 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:10:11.623983 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5015c227_7a33_4970_86c9_a1a88274a726.slice/crio-4c22bfb612350fe7cc54f0dfca868d1bee1443c525d8d2b3aeb39831ca608a7b WatchSource:0}: Error finding container 4c22bfb612350fe7cc54f0dfca868d1bee1443c525d8d2b3aeb39831ca608a7b: Status 404 returned error can't find the container with id 4c22bfb612350fe7cc54f0dfca868d1bee1443c525d8d2b3aeb39831ca608a7b Nov 02 15:10:12.166201 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:12.166166 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56fb547cfb-zdfhv" event={"ID":"5015c227-7a33-4970-86c9-a1a88274a726","Type":"ContainerStarted","Data":"4c22bfb612350fe7cc54f0dfca868d1bee1443c525d8d2b3aeb39831ca608a7b"} Nov 02 15:10:15.181797 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:15.181756 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56fb547cfb-zdfhv" event={"ID":"5015c227-7a33-4970-86c9-a1a88274a726","Type":"ContainerStarted","Data":"fcabf32fe771fee8308ecfca4435ba1e955ef5b1574956fe08aa8c1e53756df5"} Nov 02 15:10:15.292964 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:15.292899 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56fb547cfb-zdfhv" podStartSLOduration=2.5093735219999997 podStartE2EDuration="5.292878249s" totalImagesPullingTime="2.783504727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:15.291337847 +0000 UTC m=+717.494215906" watchObservedRunningTime="2026-11-02 15:10:15.292878249 +0000 UTC m=+717.495756386" Nov 02 15:10:18.478424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.478371 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f"] Nov 02 15:10:18.482243 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.482222 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.581820 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.581788 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f"] Nov 02 15:10:18.630228 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.630186 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.630397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.630266 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7fh9\" (UniqueName: \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-kube-api-access-g7fh9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.630397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.630326 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.630397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.630350 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.630397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.630377 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.630397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.630394 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.630576 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.630524 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.630576 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.630565 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.630657 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.630641 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.731813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.731731 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fh9\" (UniqueName: \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-kube-api-access-g7fh9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.731813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.731782 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.731817 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.731865 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.731894 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.731942 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.731986 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732301 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.732086 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732301 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.732129 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.732489 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732599 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.732552 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.732667 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732751 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.732715 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.732905 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.732888 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.734377 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.734350 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.734478 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.734453 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.789410 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.789375 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7fh9\" (UniqueName: \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-kube-api-access-g7fh9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.789549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.789371 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x665f\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:18.798278 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:18.798238 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:19.222853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:19.222821 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f"] Nov 02 15:10:19.228334 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:10:19.228309 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493aa9f5_503c_4790_9f0d_d55c7f9757f1.slice/crio-ddee8adff5f8917c2f973abfbcbdf5a24697c77701072ee96ca5c9ad9b4e8797 WatchSource:0}: Error finding container ddee8adff5f8917c2f973abfbcbdf5a24697c77701072ee96ca5c9ad9b4e8797: Status 404 returned error can't find the container with id ddee8adff5f8917c2f973abfbcbdf5a24697c77701072ee96ca5c9ad9b4e8797 Nov 02 15:10:19.230466 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:19.230437 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:10:19.230545 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:19.230497 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:10:19.230545 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:19.230524 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:10:20.147585 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:10:20.147549 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:10:20.208221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:20.208184 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" event={"ID":"493aa9f5-503c-4790-9f0d-d55c7f9757f1","Type":"ContainerStarted","Data":"32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396"} Nov 02 15:10:20.208414 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:20.208250 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" event={"ID":"493aa9f5-503c-4790-9f0d-d55c7f9757f1","Type":"ContainerStarted","Data":"ddee8adff5f8917c2f973abfbcbdf5a24697c77701072ee96ca5c9ad9b4e8797"} Nov 02 15:10:20.393288 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:20.393215 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" podStartSLOduration=2.393191187 podStartE2EDuration="2.393191187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:20.371430967 +0000 UTC m=+722.574309028" watchObservedRunningTime="2026-11-02 15:10:20.393191187 +0000 UTC m=+722.596069248" Nov 02 15:10:20.798975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:20.798929 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:20.800822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:20.800785 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x665f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 15:10:20.800938 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:20.800839 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 15:10:21.798730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:21.798693 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x665f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 15:10:21.799263 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:21.798760 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 15:10:22.801016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:22.800951 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x665f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 15:10:22.801614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:22.801580 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 15:10:23.799221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:23.799180 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x665f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 15:10:23.799411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:23.799239 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 15:10:24.799502 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:24.799460 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x665f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 15:10:24.799897 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:24.799531 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 15:10:25.799635 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:25.799602 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x665f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 15:10:25.800035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:25.799655 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 15:10:27.799453 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:27.799351 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x665f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:10:27.799453 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:27.799417 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:10:28.840199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:28.840155 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x665f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:10:28.840663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:28.840220 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:10:29.501894 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:29.501859 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:29.502224 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:29.502196 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:29.502912 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:10:29.502890 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:10:50.146094 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:10:50.146056 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:11:20.145926 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:11:20.145892 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:11:23.211820 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.211783 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm"] Nov 02 15:11:23.215890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.215865 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.248632 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.248608 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.248758 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.248648 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.248758 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.248672 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.248878 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.248852 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.248985 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.248958 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.249093 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.249076 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.249203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.249117 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.249253 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.249217 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6psl\" (UniqueName: \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-kube-api-access-n6psl\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.249301 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.249271 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.281161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.281141 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8a2e6f-openshift-default-dockercfg-sqgd8\"" Nov 02 15:11:23.329038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.329017 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm"] Nov 02 15:11:23.350646 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.350601 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.350796 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.350667 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.350796 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.350700 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.350796 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.350734 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.350796 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.350777 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6psl\" (UniqueName: \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-kube-api-access-n6psl\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.351056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.350826 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.351056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.350860 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.351056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.350901 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.351056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.350932 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.351261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.351211 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.351329 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.351263 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.351559 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.351497 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.351723 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.351705 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.351799 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.351777 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.353792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.353766 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.353872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.353800 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.415955 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.415927 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.416116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.415975 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6psl\" (UniqueName: \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-kube-api-access-n6psl\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.529498 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.529422 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:23.955540 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.955510 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw"] Nov 02 15:11:23.960480 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:23.960454 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.012916 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.012880 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-891723-openshift-default-dockercfg-c9wsk\"" Nov 02 15:11:24.017237 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.017208 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm"] Nov 02 15:11:24.021052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.019862 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw"] Nov 02 15:11:24.023032 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:11:24.022964 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8714e1b_0f48_4e13_afb7_9b9970b88cb3.slice/crio-b257075ded39b7d121f3fd90c6c7e240fd0925aa775dc7b8aad3ef6019d4f52a WatchSource:0}: Error finding container b257075ded39b7d121f3fd90c6c7e240fd0925aa775dc7b8aad3ef6019d4f52a: Status 404 returned error can't find the container with id b257075ded39b7d121f3fd90c6c7e240fd0925aa775dc7b8aad3ef6019d4f52a Nov 02 15:11:24.026063 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.025933 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:24.026063 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.026023 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:24.026224 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.026067 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:24.057903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.057881 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.058021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.057912 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.058074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.058033 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.058126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.058074 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.058164 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.058127 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.058202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.058191 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.058244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.058228 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnphw\" (UniqueName: \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-kube-api-access-mnphw\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.058295 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.058250 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.058295 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.058265 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.159289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.159252 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.159289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.159292 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.159565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.159323 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.159565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.159351 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.159565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.159395 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.159565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.159429 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.159565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.159476 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.159565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.159507 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.159882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.159566 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnphw\" (UniqueName: \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-kube-api-access-mnphw\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.159882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.159716 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.160086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.159974 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.160596 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.160573 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.160652 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.160593 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.161106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.161082 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.162126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.162108 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.162228 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.162155 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.219901 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.219834 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnphw\" (UniqueName: \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-kube-api-access-mnphw\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.220248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.220173 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.269500 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.269468 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq"] Nov 02 15:11:24.274092 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.274076 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.281770 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.281410 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:24.326193 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.326163 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-61a835-openshift-default-dockercfg-pq94w\"" Nov 02 15:11:24.337449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.336591 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq"] Nov 02 15:11:24.360815 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.360787 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.360958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.360835 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037fbba8-633a-4c23-bd5e-d138c3c85732-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.360958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.360866 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngbc\" (UniqueName: \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-kube-api-access-wngbc\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.360958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.360944 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.361160 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.361039 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.361160 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.361070 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.361160 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.361104 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.361160 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.361142 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.361355 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.361165 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462201 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462167 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462201 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462213 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462242 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462273 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462295 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462341 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462369 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037fbba8-633a-4c23-bd5e-d138c3c85732-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462399 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngbc\" (UniqueName: \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-kube-api-access-wngbc\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462481 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462640 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462741 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462778 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.462973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.462845 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.463221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.463204 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037fbba8-633a-4c23-bd5e-d138c3c85732-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.465122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.465099 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.465234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.465218 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.536268 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.536178 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" event={"ID":"b8714e1b-0f48-4e13-afb7-9b9970b88cb3","Type":"ContainerStarted","Data":"cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f"} Nov 02 15:11:24.536268 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.536234 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" event={"ID":"b8714e1b-0f48-4e13-afb7-9b9970b88cb3","Type":"ContainerStarted","Data":"b257075ded39b7d121f3fd90c6c7e240fd0925aa775dc7b8aad3ef6019d4f52a"} Nov 02 15:11:24.547952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.547922 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngbc\" (UniqueName: \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-kube-api-access-wngbc\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.548561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.548544 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.587817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.587780 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:24.663393 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.663331 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" podStartSLOduration=1.663311338 podStartE2EDuration="1.663311338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:24.660123184 +0000 UTC m=+786.863001246" watchObservedRunningTime="2026-11-02 15:11:24.663311338 +0000 UTC m=+786.866189398" Nov 02 15:11:24.700705 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.700673 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw"] Nov 02 15:11:24.706865 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:11:24.706831 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a006dc5_eb85_4bf5_8e94_a8b5b3fe4294.slice/crio-a15ca2381eb07959df570eb5716511791c3e8b1446b9fde1ae99c2e79c180b6d WatchSource:0}: Error finding container a15ca2381eb07959df570eb5716511791c3e8b1446b9fde1ae99c2e79c180b6d: Status 404 returned error can't find the container with id a15ca2381eb07959df570eb5716511791c3e8b1446b9fde1ae99c2e79c180b6d Nov 02 15:11:24.709256 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.709226 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:24.709337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.709288 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:24.709337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.709318 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:24.964182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.964145 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq"] Nov 02 15:11:24.968958 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:11:24.968928 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037fbba8_633a_4c23_bd5e_d138c3c85732.slice/crio-642bc4e2f05b4ae637f5d27555772cafaad8f8bdca1443b236eec757cddb12ab WatchSource:0}: Error finding container 642bc4e2f05b4ae637f5d27555772cafaad8f8bdca1443b236eec757cddb12ab: Status 404 returned error can't find the container with id 642bc4e2f05b4ae637f5d27555772cafaad8f8bdca1443b236eec757cddb12ab Nov 02 15:11:24.971218 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.971187 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:24.971299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.971251 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:24.971299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:24.971288 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:25.530525 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:25.530488 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:25.535591 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:25.535563 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:25.542439 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:25.542409 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" event={"ID":"037fbba8-633a-4c23-bd5e-d138c3c85732","Type":"ContainerStarted","Data":"18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52"} Nov 02 15:11:25.542595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:25.542466 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" event={"ID":"037fbba8-633a-4c23-bd5e-d138c3c85732","Type":"ContainerStarted","Data":"642bc4e2f05b4ae637f5d27555772cafaad8f8bdca1443b236eec757cddb12ab"} Nov 02 15:11:25.544020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:25.543974 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" event={"ID":"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294","Type":"ContainerStarted","Data":"992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad"} Nov 02 15:11:25.544111 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:25.544046 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" event={"ID":"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294","Type":"ContainerStarted","Data":"a15ca2381eb07959df570eb5716511791c3e8b1446b9fde1ae99c2e79c180b6d"} Nov 02 15:11:25.544282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:25.544266 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:25.545231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:25.545213 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:25.588645 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:25.588613 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:25.890443 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:25.890389 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" podStartSLOduration=2.890372804 podStartE2EDuration="2.890372804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:25.889265004 +0000 UTC m=+788.092143066" watchObservedRunningTime="2026-11-02 15:11:25.890372804 +0000 UTC m=+788.093250864" Nov 02 15:11:26.012241 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:26.012183 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" podStartSLOduration=2.012168129 podStartE2EDuration="2.012168129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:26.010448247 +0000 UTC m=+788.213326308" watchObservedRunningTime="2026-11-02 15:11:26.012168129 +0000 UTC m=+788.215046189" Nov 02 15:11:26.282855 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:26.282774 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:26.287693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:26.287668 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:26.550511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:26.550418 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:26.551289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:26.551265 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:11:26.593230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:26.593201 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:27.555056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:27.555016 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:27.555906 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:27.555880 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:11:36.410544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.410505 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt"] Nov 02 15:11:36.414452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.414428 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:11:36.464918 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.464884 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8a2e6f\"" Nov 02 15:11:36.485479 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.485425 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d945fed9-7518-4fd5-bb42-ef6d721c1aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt\" (UID: \"d945fed9-7518-4fd5-bb42-ef6d721c1aba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:11:36.485669 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.485543 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zv2\" (UniqueName: \"kubernetes.io/projected/d945fed9-7518-4fd5-bb42-ef6d721c1aba-kube-api-access-x8zv2\") pod \"payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt\" (UID: \"d945fed9-7518-4fd5-bb42-ef6d721c1aba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:11:36.511116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.511088 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8a2e6f-dockercfg-kcwxl\"" Nov 02 15:11:36.513152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.513125 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt"] Nov 02 15:11:36.559319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.559292 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl"] Nov 02 15:11:36.563019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.562987 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:11:36.587200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.586732 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d945fed9-7518-4fd5-bb42-ef6d721c1aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt\" (UID: \"d945fed9-7518-4fd5-bb42-ef6d721c1aba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:11:36.587200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.586833 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zv2\" (UniqueName: \"kubernetes.io/projected/d945fed9-7518-4fd5-bb42-ef6d721c1aba-kube-api-access-x8zv2\") pod \"payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt\" (UID: \"d945fed9-7518-4fd5-bb42-ef6d721c1aba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:11:36.588012 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.587976 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d945fed9-7518-4fd5-bb42-ef6d721c1aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt\" (UID: \"d945fed9-7518-4fd5-bb42-ef6d721c1aba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:11:36.589018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.588969 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl"] Nov 02 15:11:36.644599 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.644570 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zv2\" (UniqueName: \"kubernetes.io/projected/d945fed9-7518-4fd5-bb42-ef6d721c1aba-kube-api-access-x8zv2\") pod \"payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt\" (UID: \"d945fed9-7518-4fd5-bb42-ef6d721c1aba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:11:36.687822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.687761 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r586\" (UniqueName: \"kubernetes.io/projected/f3570096-5eee-4999-ae44-e3386b5e96f3-kube-api-access-8r586\") pod \"payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl\" (UID: \"f3570096-5eee-4999-ae44-e3386b5e96f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:11:36.687949 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.687824 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3570096-5eee-4999-ae44-e3386b5e96f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl\" (UID: \"f3570096-5eee-4999-ae44-e3386b5e96f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:11:36.727857 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.727832 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:11:36.788967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.788611 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r586\" (UniqueName: \"kubernetes.io/projected/f3570096-5eee-4999-ae44-e3386b5e96f3-kube-api-access-8r586\") pod \"payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl\" (UID: \"f3570096-5eee-4999-ae44-e3386b5e96f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:11:36.788967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.788702 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3570096-5eee-4999-ae44-e3386b5e96f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl\" (UID: \"f3570096-5eee-4999-ae44-e3386b5e96f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:11:36.789659 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.789635 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3570096-5eee-4999-ae44-e3386b5e96f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl\" (UID: \"f3570096-5eee-4999-ae44-e3386b5e96f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:11:36.847434 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.847402 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r586\" (UniqueName: \"kubernetes.io/projected/f3570096-5eee-4999-ae44-e3386b5e96f3-kube-api-access-8r586\") pod \"payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl\" (UID: \"f3570096-5eee-4999-ae44-e3386b5e96f3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:11:36.876184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:36.876151 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:11:37.102959 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:37.102892 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt"] Nov 02 15:11:37.108471 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:11:37.108444 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd945fed9_7518_4fd5_bb42_ef6d721c1aba.slice/crio-1773bf7bc27c35646929b1331d4a3ecd2db84ed5b6aacc6202b8f0804c517b1d WatchSource:0}: Error finding container 1773bf7bc27c35646929b1331d4a3ecd2db84ed5b6aacc6202b8f0804c517b1d: Status 404 returned error can't find the container with id 1773bf7bc27c35646929b1331d4a3ecd2db84ed5b6aacc6202b8f0804c517b1d Nov 02 15:11:37.252300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:37.252268 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl"] Nov 02 15:11:37.256690 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:11:37.256666 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3570096_5eee_4999_ae44_e3386b5e96f3.slice/crio-7398eec0178bb44cb7a5bece8fbcaeb8d5ffbd1219641f8b22cc427c1cae109f WatchSource:0}: Error finding container 7398eec0178bb44cb7a5bece8fbcaeb8d5ffbd1219641f8b22cc427c1cae109f: Status 404 returned error can't find the container with id 7398eec0178bb44cb7a5bece8fbcaeb8d5ffbd1219641f8b22cc427c1cae109f Nov 02 15:11:37.611559 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:37.611527 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" event={"ID":"f3570096-5eee-4999-ae44-e3386b5e96f3","Type":"ContainerStarted","Data":"7398eec0178bb44cb7a5bece8fbcaeb8d5ffbd1219641f8b22cc427c1cae109f"} Nov 02 15:11:37.612651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:37.612623 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" event={"ID":"d945fed9-7518-4fd5-bb42-ef6d721c1aba","Type":"ContainerStarted","Data":"1773bf7bc27c35646929b1331d4a3ecd2db84ed5b6aacc6202b8f0804c517b1d"} Nov 02 15:11:39.018034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.000433 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d"] Nov 02 15:11:39.018034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.006683 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:11:39.058823 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.058794 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-fce776\"" Nov 02 15:11:39.104714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.104686 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-fce776-dockercfg-ssxml\"" Nov 02 15:11:39.107548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.107523 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d"] Nov 02 15:11:39.119599 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.119572 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gdbq\" (UniqueName: \"kubernetes.io/projected/4829d867-65f5-4784-a4fb-95d488ac9bcf-kube-api-access-4gdbq\") pod \"payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d\" (UID: \"4829d867-65f5-4784-a4fb-95d488ac9bcf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:11:39.119728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.119623 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4829d867-65f5-4784-a4fb-95d488ac9bcf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d\" (UID: \"4829d867-65f5-4784-a4fb-95d488ac9bcf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:11:39.220399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.220364 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gdbq\" (UniqueName: \"kubernetes.io/projected/4829d867-65f5-4784-a4fb-95d488ac9bcf-kube-api-access-4gdbq\") pod \"payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d\" (UID: \"4829d867-65f5-4784-a4fb-95d488ac9bcf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:11:39.220567 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.220424 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4829d867-65f5-4784-a4fb-95d488ac9bcf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d\" (UID: \"4829d867-65f5-4784-a4fb-95d488ac9bcf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:11:39.221410 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.221389 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4829d867-65f5-4784-a4fb-95d488ac9bcf-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d\" (UID: \"4829d867-65f5-4784-a4fb-95d488ac9bcf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:11:39.279211 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.279145 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gdbq\" (UniqueName: \"kubernetes.io/projected/4829d867-65f5-4784-a4fb-95d488ac9bcf-kube-api-access-4gdbq\") pod \"payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d\" (UID: \"4829d867-65f5-4784-a4fb-95d488ac9bcf\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:11:39.334121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.334086 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:11:39.732650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:39.732611 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d"] Nov 02 15:11:39.737893 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:11:39.737866 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4829d867_65f5_4784_a4fb_95d488ac9bcf.slice/crio-d06f86531c9e3cf836827dac6380d00adecedc68256e97102b40cbc6d9497811 WatchSource:0}: Error finding container d06f86531c9e3cf836827dac6380d00adecedc68256e97102b40cbc6d9497811: Status 404 returned error can't find the container with id d06f86531c9e3cf836827dac6380d00adecedc68256e97102b40cbc6d9497811 Nov 02 15:11:40.634341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:40.634311 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" event={"ID":"f3570096-5eee-4999-ae44-e3386b5e96f3","Type":"ContainerStarted","Data":"4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270"} Nov 02 15:11:40.634684 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:40.634374 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:11:40.635767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:40.635746 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" event={"ID":"d945fed9-7518-4fd5-bb42-ef6d721c1aba","Type":"ContainerStarted","Data":"913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670"} Nov 02 15:11:40.635861 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:40.635839 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:11:40.636903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:40.636880 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" event={"ID":"4829d867-65f5-4784-a4fb-95d488ac9bcf","Type":"ContainerStarted","Data":"d06f86531c9e3cf836827dac6380d00adecedc68256e97102b40cbc6d9497811"} Nov 02 15:11:40.747632 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:40.747584 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" podStartSLOduration=1.726896989 podStartE2EDuration="4.747570216s" totalImagesPullingTime="3.020673227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:40.7452463 +0000 UTC m=+802.948124360" watchObservedRunningTime="2026-11-02 15:11:40.747570216 +0000 UTC m=+802.950448276" Nov 02 15:11:40.869359 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:40.869262 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" podStartSLOduration=1.662925585 podStartE2EDuration="4.869246926s" totalImagesPullingTime="3.206321341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:40.867365969 +0000 UTC m=+803.070244031" watchObservedRunningTime="2026-11-02 15:11:40.869246926 +0000 UTC m=+803.072124990" Nov 02 15:11:41.644329 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:41.644296 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" event={"ID":"4829d867-65f5-4784-a4fb-95d488ac9bcf","Type":"ContainerStarted","Data":"9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d"} Nov 02 15:11:41.757925 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:41.757877 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" podStartSLOduration=2.892369209 podStartE2EDuration="3.757864062s" totalImagesPullingTime="865.494853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:41.755665635 +0000 UTC m=+803.958543692" watchObservedRunningTime="2026-11-02 15:11:41.757864062 +0000 UTC m=+803.960742122" Nov 02 15:11:42.308135 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:42.308100 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:11:42.650482 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:42.650449 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:11:50.146850 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:11:50.146805 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:11:51.104084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.104053 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:11:51.107176 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.107143 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j"] Nov 02 15:11:51.111279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.111251 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.167717 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.167679 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-maas-api-tls\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.168166 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.167799 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb839896-6a9a-4d11-82a6-597c9d569411-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.168166 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.167843 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgnzh\" (UniqueName: \"kubernetes.io/projected/eb839896-6a9a-4d11-82a6-597c9d569411-kube-api-access-bgnzh\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.168166 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.167924 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-metrics-certs\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.189051 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.178265 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j"] Nov 02 15:11:51.229847 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.229826 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-e98976-serving-cert\"" Nov 02 15:11:51.230050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.230035 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 15:11:51.230111 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.230041 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wj8b6\"" Nov 02 15:11:51.269255 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.269226 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb839896-6a9a-4d11-82a6-597c9d569411-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.269386 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.269267 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgnzh\" (UniqueName: \"kubernetes.io/projected/eb839896-6a9a-4d11-82a6-597c9d569411-kube-api-access-bgnzh\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.269386 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.269299 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-metrics-certs\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.269386 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.269357 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-maas-api-tls\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.269503 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:11:51.269483 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-serving-cert: secret "maas-api-e2e-worker-w6-e98976-serving-cert" not found Nov 02 15:11:51.269561 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:11:51.269551 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-maas-api-tls podName:eb839896-6a9a-4d11-82a6-597c9d569411 nodeName:}" failed. No retries permitted until 2026-11-02 15:11:51.769522776 +0000 UTC m=+813.972400827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-maas-api-tls") pod "maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" (UID: "eb839896-6a9a-4d11-82a6-597c9d569411") : secret "maas-api-e2e-worker-w6-e98976-serving-cert" not found Nov 02 15:11:51.271824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.271795 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-metrics-certs\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.275706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.275677 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:11:51.276194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.276169 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:11:51.280095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.280075 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb839896-6a9a-4d11-82a6-597c9d569411-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.330152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.330125 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgnzh\" (UniqueName: \"kubernetes.io/projected/eb839896-6a9a-4d11-82a6-597c9d569411-kube-api-access-bgnzh\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.654014 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.653426 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:11:51.660014 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.658142 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:11:51.737600 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.737562 5476 generic.go:356] "Generic (PLEG): container finished" podID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerID="cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f" exitCode=0 Nov 02 15:11:51.737775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.737628 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" event={"ID":"b8714e1b-0f48-4e13-afb7-9b9970b88cb3","Type":"ContainerDied","Data":"cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f"} Nov 02 15:11:51.738086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.738067 5476 scope.go:122] "RemoveContainer" containerID="cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f" Nov 02 15:11:51.739700 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.739668 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:51.739774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.739757 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:51.739814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.739804 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:11:51.779446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.779413 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-maas-api-tls\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:51.782869 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:51.782806 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-maas-api-tls\") pod \"maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:52.059170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:52.059082 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:52.433906 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:52.433870 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j"] Nov 02 15:11:52.441397 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:11:52.441365 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb839896_6a9a_4d11_82a6_597c9d569411.slice/crio-3f866d7494a31c0be1ae9811a9ad37b4272ed02f28719653fcdf0234a9153b2e WatchSource:0}: Error finding container 3f866d7494a31c0be1ae9811a9ad37b4272ed02f28719653fcdf0234a9153b2e: Status 404 returned error can't find the container with id 3f866d7494a31c0be1ae9811a9ad37b4272ed02f28719653fcdf0234a9153b2e Nov 02 15:11:52.744930 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:52.744842 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" event={"ID":"b8714e1b-0f48-4e13-afb7-9b9970b88cb3","Type":"ContainerStarted","Data":"44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0"} Nov 02 15:11:52.746238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:52.746215 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" event={"ID":"eb839896-6a9a-4d11-82a6-597c9d569411","Type":"ContainerStarted","Data":"3f866d7494a31c0be1ae9811a9ad37b4272ed02f28719653fcdf0234a9153b2e"} Nov 02 15:11:53.530526 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:53.530489 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:53.531852 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:53.531824 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 15:11:53.531969 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:53.531878 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 15:11:53.662034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:53.661984 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:11:54.530616 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:54.530581 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 15:11:54.531092 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:54.530635 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 15:11:55.530435 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:55.530357 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 15:11:55.530435 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:55.530409 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 15:11:55.544533 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:55.544504 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:55.999340 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:55.999291 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7"] Nov 02 15:11:56.003697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.003674 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:11:56.031498 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.031470 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkrzs\" (UniqueName: \"kubernetes.io/projected/9f01d442-c187-4005-88e0-e36e9cbabc5b-kube-api-access-bkrzs\") pod \"payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7\" (UID: \"9f01d442-c187-4005-88e0-e36e9cbabc5b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:11:56.031602 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.031555 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f01d442-c187-4005-88e0-e36e9cbabc5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7\" (UID: \"9f01d442-c187-4005-88e0-e36e9cbabc5b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:11:56.062812 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.062772 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-61a835-dockercfg-rfdvb\"" Nov 02 15:11:56.069934 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.069908 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7"] Nov 02 15:11:56.110410 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.110363 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-61a835\"" Nov 02 15:11:56.132554 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.132508 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkrzs\" (UniqueName: \"kubernetes.io/projected/9f01d442-c187-4005-88e0-e36e9cbabc5b-kube-api-access-bkrzs\") pod \"payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7\" (UID: \"9f01d442-c187-4005-88e0-e36e9cbabc5b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:11:56.132740 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.132607 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f01d442-c187-4005-88e0-e36e9cbabc5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7\" (UID: \"9f01d442-c187-4005-88e0-e36e9cbabc5b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:11:56.134125 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.134072 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f01d442-c187-4005-88e0-e36e9cbabc5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7\" (UID: \"9f01d442-c187-4005-88e0-e36e9cbabc5b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:11:56.193018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.170057 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf"] Nov 02 15:11:56.198442 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.198419 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:11:56.200259 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.200239 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkrzs\" (UniqueName: \"kubernetes.io/projected/9f01d442-c187-4005-88e0-e36e9cbabc5b-kube-api-access-bkrzs\") pod \"payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7\" (UID: \"9f01d442-c187-4005-88e0-e36e9cbabc5b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:11:56.234015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.233913 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f947d46-4501-4a17-ae94-72793390fd4f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf\" (UID: \"7f947d46-4501-4a17-ae94-72793390fd4f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:11:56.234015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.233972 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2ptg\" (UniqueName: \"kubernetes.io/projected/7f947d46-4501-4a17-ae94-72793390fd4f-kube-api-access-n2ptg\") pod \"payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf\" (UID: \"7f947d46-4501-4a17-ae94-72793390fd4f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:11:56.243869 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.243841 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf"] Nov 02 15:11:56.322497 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.322432 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:11:56.335008 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.334913 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2ptg\" (UniqueName: \"kubernetes.io/projected/7f947d46-4501-4a17-ae94-72793390fd4f-kube-api-access-n2ptg\") pod \"payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf\" (UID: \"7f947d46-4501-4a17-ae94-72793390fd4f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:11:56.335138 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.335100 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f947d46-4501-4a17-ae94-72793390fd4f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf\" (UID: \"7f947d46-4501-4a17-ae94-72793390fd4f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:11:56.335899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.335873 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f947d46-4501-4a17-ae94-72793390fd4f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf\" (UID: \"7f947d46-4501-4a17-ae94-72793390fd4f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:11:56.405215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.403752 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2ptg\" (UniqueName: \"kubernetes.io/projected/7f947d46-4501-4a17-ae94-72793390fd4f-kube-api-access-n2ptg\") pod \"payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf\" (UID: \"7f947d46-4501-4a17-ae94-72793390fd4f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:11:56.522960 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.522937 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:11:56.530289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.530266 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 15:11:56.530386 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.530308 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 15:11:56.728536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.728506 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7"] Nov 02 15:11:56.734547 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:11:56.734513 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f01d442_c187_4005_88e0_e36e9cbabc5b.slice/crio-f9ea63e40c3c3cd96d2538e4c45b3e015e3a084d55d430ba24e9aa67b527aa74 WatchSource:0}: Error finding container f9ea63e40c3c3cd96d2538e4c45b3e015e3a084d55d430ba24e9aa67b527aa74: Status 404 returned error can't find the container with id f9ea63e40c3c3cd96d2538e4c45b3e015e3a084d55d430ba24e9aa67b527aa74 Nov 02 15:11:56.797843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.797800 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" event={"ID":"eb839896-6a9a-4d11-82a6-597c9d569411","Type":"ContainerStarted","Data":"873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691"} Nov 02 15:11:56.798095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.798070 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:11:56.799394 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.799361 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" event={"ID":"9f01d442-c187-4005-88e0-e36e9cbabc5b","Type":"ContainerStarted","Data":"f9ea63e40c3c3cd96d2538e4c45b3e015e3a084d55d430ba24e9aa67b527aa74"} Nov 02 15:11:56.912348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.912291 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" podStartSLOduration=1.838778985 podStartE2EDuration="5.912277146s" totalImagesPullingTime="4.073498161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:56.909499544 +0000 UTC m=+819.112377604" watchObservedRunningTime="2026-11-02 15:11:56.912277146 +0000 UTC m=+819.115155205" Nov 02 15:11:56.920338 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:56.920312 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf"] Nov 02 15:11:56.926605 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:11:56.926580 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f947d46_4501_4a17_ae94_72793390fd4f.slice/crio-91891acf265b855ce5027df1263fc4c8c1c37d9ecdf940bd201b7272d03d0ae7 WatchSource:0}: Error finding container 91891acf265b855ce5027df1263fc4c8c1c37d9ecdf940bd201b7272d03d0ae7: Status 404 returned error can't find the container with id 91891acf265b855ce5027df1263fc4c8c1c37d9ecdf940bd201b7272d03d0ae7 Nov 02 15:11:57.530016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:57.529967 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 02 15:11:57.530205 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:57.530042 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 02 15:11:57.808316 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:57.808206 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" event={"ID":"7f947d46-4501-4a17-ae94-72793390fd4f","Type":"ContainerStarted","Data":"47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632"} Nov 02 15:11:57.808316 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:57.808275 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" event={"ID":"7f947d46-4501-4a17-ae94-72793390fd4f","Type":"ContainerStarted","Data":"91891acf265b855ce5027df1263fc4c8c1c37d9ecdf940bd201b7272d03d0ae7"} Nov 02 15:11:57.808808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:57.808330 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:11:57.810152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:57.810118 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" event={"ID":"9f01d442-c187-4005-88e0-e36e9cbabc5b","Type":"ContainerStarted","Data":"f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9"} Nov 02 15:11:57.810283 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:57.810184 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:11:57.921592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:57.921535 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" podStartSLOduration=1.9215206120000001 podStartE2EDuration="1.921520612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:57.920304724 +0000 UTC m=+820.123182785" watchObservedRunningTime="2026-11-02 15:11:57.921520612 +0000 UTC m=+820.124398672" Nov 02 15:11:58.533935 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:58.533906 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:58.535046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:58.535023 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:11:58.695319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:11:58.695261 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" podStartSLOduration=3.695247139 podStartE2EDuration="3.695247139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:58.034345446 +0000 UTC m=+820.237223507" watchObservedRunningTime="2026-11-02 15:11:58.695247139 +0000 UTC m=+820.898125199" Nov 02 15:12:03.821354 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:03.821326 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:12:08.819351 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:08.819320 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:12:08.819856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:08.819394 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:12:10.501932 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:10.501893 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:12:18.909064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:18.905984 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:12:20.153047 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:12:20.153008 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:12:20.949636 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:20.949603 5476 generic.go:356] "Generic (PLEG): container finished" podID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerID="992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad" exitCode=0 Nov 02 15:12:20.949810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:20.949676 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" event={"ID":"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294","Type":"ContainerDied","Data":"992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad"} Nov 02 15:12:20.950071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:20.950057 5476 scope.go:122] "RemoveContainer" containerID="992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad" Nov 02 15:12:20.951233 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:20.951207 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:20.951310 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:20.951281 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:20.951347 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:20.951310 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:21.956972 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:21.956934 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" event={"ID":"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294","Type":"ContainerStarted","Data":"f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c"} Nov 02 15:12:22.293080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:22.290366 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:12:22.293080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:22.291874 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 15:12:22.293080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:22.291924 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 15:12:23.287019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:23.283456 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 15:12:23.287019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:23.283523 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 15:12:24.284428 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:24.284390 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 15:12:24.284701 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:24.284673 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 15:12:25.286018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:25.285347 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 15:12:25.286018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:25.285405 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 15:12:26.286465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:26.286422 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 15:12:26.287381 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:26.287345 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 15:12:26.552351 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:26.552264 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:12:27.290281 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:27.290247 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:12:27.291344 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:27.291321 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:12:28.597221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:28.597181 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:12:28.597780 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:28.597278 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:12:29.030161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:29.030079 5476 generic.go:356] "Generic (PLEG): container finished" podID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerID="18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52" exitCode=0 Nov 02 15:12:29.030161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:29.030119 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" event={"ID":"037fbba8-633a-4c23-bd5e-d138c3c85732","Type":"ContainerDied","Data":"18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52"} Nov 02 15:12:29.030570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:29.030553 5476 scope.go:122] "RemoveContainer" containerID="18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52" Nov 02 15:12:29.031972 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:29.031948 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:29.032076 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:29.032041 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:29.032076 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:29.032073 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:12:30.037986 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:30.037952 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" event={"ID":"037fbba8-633a-4c23-bd5e-d138c3c85732","Type":"ContainerStarted","Data":"0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865"} Nov 02 15:12:30.596247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:30.596203 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:12:30.596448 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:30.596419 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 15:12:30.596529 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:30.596473 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 15:12:31.591407 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:31.591367 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 15:12:31.592027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:31.591971 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 15:12:32.593019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:32.589352 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 15:12:32.593019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:32.589448 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 15:12:33.591017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:33.590328 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 15:12:33.591017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:33.590387 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 15:12:34.589402 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:34.589362 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 15:12:34.589988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:34.589958 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 15:12:35.592098 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:35.592067 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:12:35.592511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:35.592458 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:12:35.593128 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:35.593107 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:12:36.709106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:12:36.709060 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:12:50.146015 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:12:50.145956 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:13:00.488460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:00.488421 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:13:08.662785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.662749 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm"] Nov 02 15:13:08.666621 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.666604 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.717620 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.717571 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"simulator-w6-e98976-kserve-self-signed-certs\"" Nov 02 15:13:08.747170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.747138 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-dshm\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.747170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.747176 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-home\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.747376 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.747201 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ffe407-4378-44c8-846d-fafc02f5d26f-tls-certs\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.747376 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.747324 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrqj\" (UniqueName: \"kubernetes.io/projected/f5ffe407-4378-44c8-846d-fafc02f5d26f-kube-api-access-hsrqj\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.747500 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.747477 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-model-cache\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.747558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.747535 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-tmp-dir\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.763889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.763867 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"default-dockercfg-xk8r7\"" Nov 02 15:13:08.764038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.763930 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"openshift-service-ca.crt\"" Nov 02 15:13:08.764144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.764127 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"kube-root-ca.crt\"" Nov 02 15:13:08.766320 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.766302 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm"] Nov 02 15:13:08.848073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.848044 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-model-cache\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.848254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.848085 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-tmp-dir\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.848254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.848115 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-dshm\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.848254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.848138 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-home\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.848254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.848164 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ffe407-4378-44c8-846d-fafc02f5d26f-tls-certs\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.848254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.848188 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrqj\" (UniqueName: \"kubernetes.io/projected/f5ffe407-4378-44c8-846d-fafc02f5d26f-kube-api-access-hsrqj\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.848538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.848517 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-tmp-dir\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.848584 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.848554 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-home\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.848584 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.848573 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-model-cache\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.850464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.850440 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-dshm\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.850672 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.850655 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ffe407-4378-44c8-846d-fafc02f5d26f-tls-certs\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.905695 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.905665 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrqj\" (UniqueName: \"kubernetes.io/projected/f5ffe407-4378-44c8-846d-fafc02f5d26f-kube-api-access-hsrqj\") pod \"simulator-w6-e98976-kserve-8645fc65b5-sc2pm\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:08.978467 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:08.978391 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:09.348683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:09.348606 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm"] Nov 02 15:13:09.354420 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:13:09.354396 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5ffe407_4378_44c8_846d_fafc02f5d26f.slice/crio-568666776a8a49f040212e9cebaef583b50cc8cdb68569f06578fb16581990c1 WatchSource:0}: Error finding container 568666776a8a49f040212e9cebaef583b50cc8cdb68569f06578fb16581990c1: Status 404 returned error can't find the container with id 568666776a8a49f040212e9cebaef583b50cc8cdb68569f06578fb16581990c1 Nov 02 15:13:10.283590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:10.283556 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" event={"ID":"f5ffe407-4378-44c8-846d-fafc02f5d26f","Type":"ContainerStarted","Data":"b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042"} Nov 02 15:13:10.283968 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:10.283603 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" event={"ID":"f5ffe407-4378-44c8-846d-fafc02f5d26f","Type":"ContainerStarted","Data":"568666776a8a49f040212e9cebaef583b50cc8cdb68569f06578fb16581990c1"} Nov 02 15:13:10.398417 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:10.398364 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podStartSLOduration=2.398349868 podStartE2EDuration="2.398349868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:10.396455265 +0000 UTC m=+892.599333324" watchObservedRunningTime="2026-11-02 15:13:10.398349868 +0000 UTC m=+892.601227928" Nov 02 15:13:10.979454 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:10.979424 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:13.088210 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:13.088170 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:13:18.033012 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:18.032972 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:13:18.035566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:18.035542 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:13:18.979405 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:18.979368 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:18.992162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:18.992137 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:19.345782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:19.345709 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:13:20.146918 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:13:20.146878 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:13:33.233823 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:33.233725 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:13:33.828156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:33.828114 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz"] Nov 02 15:13:33.833168 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:33.833138 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:33.934071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:33.933989 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8a2e6f\"/\"openshift-service-ca.crt\"" Nov 02 15:13:33.934071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:33.934013 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a2e6f\"/\"premium-w4-8a2e6f-kserve-self-signed-certs\"" Nov 02 15:13:33.934071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:33.934039 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8a2e6f\"/\"kube-root-ca.crt\"" Nov 02 15:13:33.935245 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:33.935223 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a2e6f\"/\"default-dockercfg-2ssb5\"" Nov 02 15:13:33.938065 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:33.938011 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz"] Nov 02 15:13:34.009389 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.009344 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-dshm\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.009592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.009450 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kkqc\" (UniqueName: \"kubernetes.io/projected/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-kube-api-access-8kkqc\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.009592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.009483 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tmp-dir\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.009592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.009577 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-home\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.009756 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.009624 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-model-cache\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.009756 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.009666 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tls-certs\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.111319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.111285 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-dshm\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.111532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.111377 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kkqc\" (UniqueName: \"kubernetes.io/projected/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-kube-api-access-8kkqc\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.111532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.111409 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tmp-dir\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.111532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.111449 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-home\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.111711 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.111629 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-model-cache\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.111769 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.111709 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tls-certs\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.111861 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.111841 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-home\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.111935 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.111911 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tmp-dir\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.112094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.112072 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-model-cache\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.113801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.113779 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-dshm\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.114180 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.114162 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tls-certs\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.169286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.169252 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kkqc\" (UniqueName: \"kubernetes.io/projected/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-kube-api-access-8kkqc\") pod \"premium-w4-8a2e6f-kserve-c6486b896-l5rxz\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.451844 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.451758 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:34.825371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:34.825283 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz"] Nov 02 15:13:34.831122 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:13:34.831092 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dba751d_d54a_46a8_96eb_5adb4cbc66ef.slice/crio-c6799a2de72336902f53c52a3d99c93c396cf41e7f045a68af16e1f084cfecdd WatchSource:0}: Error finding container c6799a2de72336902f53c52a3d99c93c396cf41e7f045a68af16e1f084cfecdd: Status 404 returned error can't find the container with id c6799a2de72336902f53c52a3d99c93c396cf41e7f045a68af16e1f084cfecdd Nov 02 15:13:35.439741 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:35.439697 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" event={"ID":"7dba751d-d54a-46a8-96eb-5adb4cbc66ef","Type":"ContainerStarted","Data":"13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb"} Nov 02 15:13:35.439741 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:35.439745 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" event={"ID":"7dba751d-d54a-46a8-96eb-5adb4cbc66ef","Type":"ContainerStarted","Data":"c6799a2de72336902f53c52a3d99c93c396cf41e7f045a68af16e1f084cfecdd"} Nov 02 15:13:35.452059 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:35.452035 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:35.550330 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:35.550274 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podStartSLOduration=2.55025894 podStartE2EDuration="2.55025894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:35.549244793 +0000 UTC m=+917.752122854" watchObservedRunningTime="2026-11-02 15:13:35.55025894 +0000 UTC m=+917.753137000" Nov 02 15:13:44.454898 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:44.454859 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:44.481419 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:44.481391 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:44.513710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:44.513673 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:13:48.201949 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.201912 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq"] Nov 02 15:13:48.205767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.205748 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.255105 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.255068 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdk6s\" (UniqueName: \"kubernetes.io/projected/6c629e7a-d8a7-4073-ac68-4193ec218328-kube-api-access-jdk6s\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.255289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.255192 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-home\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.255289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.255258 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c629e7a-d8a7-4073-ac68-4193ec218328-tls-certs\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.255425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.255371 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-model-cache\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.255425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.255404 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-tmp-dir\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.255550 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.255531 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-dshm\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.300851 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.300817 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-624908\"/\"simulator-w1-624908-kserve-self-signed-certs\"" Nov 02 15:13:48.301075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.300958 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-624908\"/\"openshift-service-ca.crt\"" Nov 02 15:13:48.301075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.301028 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-624908\"/\"kube-root-ca.crt\"" Nov 02 15:13:48.301075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.301049 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-624908\"/\"default-dockercfg-86rz9\"" Nov 02 15:13:48.303561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.303540 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq"] Nov 02 15:13:48.357039 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.356903 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-home\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.357412 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.357356 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c629e7a-d8a7-4073-ac68-4193ec218328-tls-certs\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.357558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.357427 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-model-cache\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.357558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.357458 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-tmp-dir\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.357558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.357538 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-dshm\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.357912 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.357616 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-home\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.357912 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.357606 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdk6s\" (UniqueName: \"kubernetes.io/projected/6c629e7a-d8a7-4073-ac68-4193ec218328-kube-api-access-jdk6s\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.358335 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.358311 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-tmp-dir\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.358460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.358321 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-model-cache\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.365327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.365295 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-dshm\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.367221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.367144 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c629e7a-d8a7-4073-ac68-4193ec218328-tls-certs\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.415728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.415702 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdk6s\" (UniqueName: \"kubernetes.io/projected/6c629e7a-d8a7-4073-ac68-4193ec218328-kube-api-access-jdk6s\") pod \"simulator-w1-624908-kserve-7cc675b85-2ggnq\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.518423 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.518351 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:48.888894 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:48.888858 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq"] Nov 02 15:13:48.894417 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:13:48.894389 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c629e7a_d8a7_4073_ac68_4193ec218328.slice/crio-5be193802e5e979362069e52ca3a47302aeb3bac30aba138f879b50f1bfe6250 WatchSource:0}: Error finding container 5be193802e5e979362069e52ca3a47302aeb3bac30aba138f879b50f1bfe6250: Status 404 returned error can't find the container with id 5be193802e5e979362069e52ca3a47302aeb3bac30aba138f879b50f1bfe6250 Nov 02 15:13:49.529113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:49.529079 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" event={"ID":"6c629e7a-d8a7-4073-ac68-4193ec218328","Type":"ContainerStarted","Data":"82fd49ee96788f5bc900ef774449e3546177e7501e0fc9872563f0c01e43ed3e"} Nov 02 15:13:49.529553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:49.529121 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" event={"ID":"6c629e7a-d8a7-4073-ac68-4193ec218328","Type":"ContainerStarted","Data":"5be193802e5e979362069e52ca3a47302aeb3bac30aba138f879b50f1bfe6250"} Nov 02 15:13:49.639547 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:49.639490 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podStartSLOduration=1.639475995 podStartE2EDuration="1.639475995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:49.638907034 +0000 UTC m=+931.841785094" watchObservedRunningTime="2026-11-02 15:13:49.639475995 +0000 UTC m=+931.842354055" Nov 02 15:13:50.146767 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:13:50.146729 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:13:50.518780 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:50.518701 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:56.807392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:56.807356 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:13:58.518694 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:58.518664 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:58.530936 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:58.530911 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:13:58.590701 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:13:58.590674 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:14:20.147625 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:14:20.147588 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:14:20.583534 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:20.583456 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:14:24.968562 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:24.968531 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf"] Nov 02 15:14:24.973919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:24.973898 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.018784 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.018759 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-tmp-dir\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.018905 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.018792 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/015490c4-4394-487f-90ff-15b5f189b583-tls-certs\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.018983 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.018957 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-model-cache\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.019057 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.019038 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-home\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.019104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.019092 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s4gv\" (UniqueName: \"kubernetes.io/projected/015490c4-4394-487f-90ff-15b5f189b583-kube-api-access-5s4gv\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.019239 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.019222 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-dshm\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.022947 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.022921 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"kube-root-ca.crt\"" Nov 02 15:14:25.049941 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.049918 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"openshift-service-ca.crt\"" Nov 02 15:14:25.068681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.068657 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"default-dockercfg-z5s42\"" Nov 02 15:14:25.068792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.068781 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"premium-w3-71d1eb-kserve-self-signed-certs\"" Nov 02 15:14:25.071380 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.071362 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf"] Nov 02 15:14:25.120583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.120549 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-dshm\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.120881 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.120857 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-tmp-dir\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.121065 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.121035 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/015490c4-4394-487f-90ff-15b5f189b583-tls-certs\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.121205 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.121183 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-model-cache\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.121283 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.121221 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-home\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.121283 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.121246 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s4gv\" (UniqueName: \"kubernetes.io/projected/015490c4-4394-487f-90ff-15b5f189b583-kube-api-access-5s4gv\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.122651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.122606 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-model-cache\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.122918 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.122896 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-home\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.123167 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.123146 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-tmp-dir\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.125780 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.125759 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-dshm\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.126545 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.126520 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/015490c4-4394-487f-90ff-15b5f189b583-tls-certs\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.178613 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.178593 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s4gv\" (UniqueName: \"kubernetes.io/projected/015490c4-4394-487f-90ff-15b5f189b583-kube-api-access-5s4gv\") pod \"premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.285604 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.285545 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:25.675652 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.675624 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf"] Nov 02 15:14:25.681078 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:14:25.681050 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod015490c4_4394_487f_90ff_15b5f189b583.slice/crio-ae4e1c12bf4894ff56a14cc417f7e44a9d48c3edea41fe3c443d75a5fc61f519 WatchSource:0}: Error finding container ae4e1c12bf4894ff56a14cc417f7e44a9d48c3edea41fe3c443d75a5fc61f519: Status 404 returned error can't find the container with id ae4e1c12bf4894ff56a14cc417f7e44a9d48c3edea41fe3c443d75a5fc61f519 Nov 02 15:14:25.738310 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.738278 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" event={"ID":"015490c4-4394-487f-90ff-15b5f189b583","Type":"ContainerStarted","Data":"ae4e1c12bf4894ff56a14cc417f7e44a9d48c3edea41fe3c443d75a5fc61f519"} Nov 02 15:14:25.850323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:25.850263 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podStartSLOduration=1.8502503670000001 podStartE2EDuration="1.850250367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:25.848061363 +0000 UTC m=+968.050939422" watchObservedRunningTime="2026-11-02 15:14:25.850250367 +0000 UTC m=+968.053128428" Nov 02 15:14:26.285834 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:26.285800 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:26.746049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:26.745986 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" event={"ID":"015490c4-4394-487f-90ff-15b5f189b583","Type":"ContainerStarted","Data":"a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a"} Nov 02 15:14:35.286261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:35.286219 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:35.299484 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:35.299452 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:35.814271 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:35.814240 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:14:49.445681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:14:49.445641 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:14:50.147045 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:14:50.147006 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:15:00.180753 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.180709 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2"] Nov 02 15:15:00.185308 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.185284 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" Nov 02 15:15:00.186142 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.186120 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq"] Nov 02 15:15:00.189780 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.189759 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q"] Nov 02 15:15:00.189922 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.189877 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" Nov 02 15:15:00.193526 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.193504 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" Nov 02 15:15:00.228425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.228394 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s"] Nov 02 15:15:00.232924 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.232899 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" Nov 02 15:15:00.235603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.235579 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:15:00.281068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.281036 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2"] Nov 02 15:15:00.296345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.295422 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73db795d-bb80-427f-bced-334e30da3621-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s\" (UID: \"73db795d-bb80-427f-bced-334e30da3621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" Nov 02 15:15:00.296345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.295503 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq\" (UID: \"bf6f0083-090a-45bc-8d6e-b42d7dd1629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" Nov 02 15:15:00.296345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.295547 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x4wt\" (UniqueName: \"kubernetes.io/projected/6ba676a7-71bc-4cce-98ea-4bd83a246df7-kube-api-access-5x4wt\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2\" (UID: \"6ba676a7-71bc-4cce-98ea-4bd83a246df7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" Nov 02 15:15:00.296345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.295605 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba676a7-71bc-4cce-98ea-4bd83a246df7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2\" (UID: \"6ba676a7-71bc-4cce-98ea-4bd83a246df7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" Nov 02 15:15:00.296345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.295656 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7ks\" (UniqueName: \"kubernetes.io/projected/e42e5418-c37e-47cc-83a6-515385294a6f-kube-api-access-fp7ks\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q\" (UID: \"e42e5418-c37e-47cc-83a6-515385294a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" Nov 02 15:15:00.296345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.295720 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sbsv\" (UniqueName: \"kubernetes.io/projected/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-kube-api-access-4sbsv\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq\" (UID: \"bf6f0083-090a-45bc-8d6e-b42d7dd1629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" Nov 02 15:15:00.296345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.295781 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmb9l\" (UniqueName: \"kubernetes.io/projected/73db795d-bb80-427f-bced-334e30da3621-kube-api-access-cmb9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s\" (UID: \"73db795d-bb80-427f-bced-334e30da3621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" Nov 02 15:15:00.296345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.295811 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42e5418-c37e-47cc-83a6-515385294a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q\" (UID: \"e42e5418-c37e-47cc-83a6-515385294a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" Nov 02 15:15:00.314953 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.307187 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq"] Nov 02 15:15:00.314953 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.307249 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q"] Nov 02 15:15:00.337299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.337262 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s"] Nov 02 15:15:00.396670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.396621 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmb9l\" (UniqueName: \"kubernetes.io/projected/73db795d-bb80-427f-bced-334e30da3621-kube-api-access-cmb9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s\" (UID: \"73db795d-bb80-427f-bced-334e30da3621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" Nov 02 15:15:00.396882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.396694 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42e5418-c37e-47cc-83a6-515385294a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q\" (UID: \"e42e5418-c37e-47cc-83a6-515385294a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" Nov 02 15:15:00.396882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.396738 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73db795d-bb80-427f-bced-334e30da3621-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s\" (UID: \"73db795d-bb80-427f-bced-334e30da3621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" Nov 02 15:15:00.396882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.396804 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq\" (UID: \"bf6f0083-090a-45bc-8d6e-b42d7dd1629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" Nov 02 15:15:00.396882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.396849 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x4wt\" (UniqueName: \"kubernetes.io/projected/6ba676a7-71bc-4cce-98ea-4bd83a246df7-kube-api-access-5x4wt\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2\" (UID: \"6ba676a7-71bc-4cce-98ea-4bd83a246df7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" Nov 02 15:15:00.397126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.396930 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba676a7-71bc-4cce-98ea-4bd83a246df7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2\" (UID: \"6ba676a7-71bc-4cce-98ea-4bd83a246df7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" Nov 02 15:15:00.397126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.396985 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7ks\" (UniqueName: \"kubernetes.io/projected/e42e5418-c37e-47cc-83a6-515385294a6f-kube-api-access-fp7ks\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q\" (UID: \"e42e5418-c37e-47cc-83a6-515385294a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" Nov 02 15:15:00.397126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.397075 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sbsv\" (UniqueName: \"kubernetes.io/projected/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-kube-api-access-4sbsv\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq\" (UID: \"bf6f0083-090a-45bc-8d6e-b42d7dd1629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" Nov 02 15:15:00.397650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.397626 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42e5418-c37e-47cc-83a6-515385294a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q\" (UID: \"e42e5418-c37e-47cc-83a6-515385294a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" Nov 02 15:15:00.397816 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.397790 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq\" (UID: \"bf6f0083-090a-45bc-8d6e-b42d7dd1629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" Nov 02 15:15:00.398238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.398213 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73db795d-bb80-427f-bced-334e30da3621-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s\" (UID: \"73db795d-bb80-427f-bced-334e30da3621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" Nov 02 15:15:00.398330 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.398288 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba676a7-71bc-4cce-98ea-4bd83a246df7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2\" (UID: \"6ba676a7-71bc-4cce-98ea-4bd83a246df7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" Nov 02 15:15:00.458248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.458175 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x4wt\" (UniqueName: \"kubernetes.io/projected/6ba676a7-71bc-4cce-98ea-4bd83a246df7-kube-api-access-5x4wt\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2\" (UID: \"6ba676a7-71bc-4cce-98ea-4bd83a246df7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" Nov 02 15:15:00.458472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.458447 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmb9l\" (UniqueName: \"kubernetes.io/projected/73db795d-bb80-427f-bced-334e30da3621-kube-api-access-cmb9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s\" (UID: \"73db795d-bb80-427f-bced-334e30da3621\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" Nov 02 15:15:00.458699 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.458675 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sbsv\" (UniqueName: \"kubernetes.io/projected/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-kube-api-access-4sbsv\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq\" (UID: \"bf6f0083-090a-45bc-8d6e-b42d7dd1629a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" Nov 02 15:15:00.459557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.459520 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7ks\" (UniqueName: \"kubernetes.io/projected/e42e5418-c37e-47cc-83a6-515385294a6f-kube-api-access-fp7ks\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q\" (UID: \"e42e5418-c37e-47cc-83a6-515385294a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" Nov 02 15:15:00.505675 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.505635 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" Nov 02 15:15:00.516533 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.516493 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" Nov 02 15:15:00.522359 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.522330 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" Nov 02 15:15:00.546835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.546576 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" Nov 02 15:15:00.897398 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.897351 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2"] Nov 02 15:15:00.906079 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:15:00.903237 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba676a7_71bc_4cce_98ea_4bd83a246df7.slice/crio-09fe224d99be4109de4db2aa5c701537e08ad85577a2e7b5bdf89c4ac98ba218 WatchSource:0}: Error finding container 09fe224d99be4109de4db2aa5c701537e08ad85577a2e7b5bdf89c4ac98ba218: Status 404 returned error can't find the container with id 09fe224d99be4109de4db2aa5c701537e08ad85577a2e7b5bdf89c4ac98ba218 Nov 02 15:15:00.909364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.909339 5476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:15:00.953825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.953789 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq"] Nov 02 15:15:00.957973 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:15:00.957946 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6f0083_090a_45bc_8d6e_b42d7dd1629a.slice/crio-1fe9392a840a4abbd02f6be481b7e4080c69e1d769cb29091faacf35cf52dbd0 WatchSource:0}: Error finding container 1fe9392a840a4abbd02f6be481b7e4080c69e1d769cb29091faacf35cf52dbd0: Status 404 returned error can't find the container with id 1fe9392a840a4abbd02f6be481b7e4080c69e1d769cb29091faacf35cf52dbd0 Nov 02 15:15:00.958348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:00.958319 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" event={"ID":"6ba676a7-71bc-4cce-98ea-4bd83a246df7","Type":"ContainerStarted","Data":"09fe224d99be4109de4db2aa5c701537e08ad85577a2e7b5bdf89c4ac98ba218"} Nov 02 15:15:01.001428 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:01.001392 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q"] Nov 02 15:15:01.003955 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:01.003930 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s"] Nov 02 15:15:01.008898 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:15:01.007172 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42e5418_c37e_47cc_83a6_515385294a6f.slice/crio-772007628c555e46e611041a5ac858cbd8a2c0b198f8ebc150fcec5b773a0633 WatchSource:0}: Error finding container 772007628c555e46e611041a5ac858cbd8a2c0b198f8ebc150fcec5b773a0633: Status 404 returned error can't find the container with id 772007628c555e46e611041a5ac858cbd8a2c0b198f8ebc150fcec5b773a0633 Nov 02 15:15:01.013804 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:15:01.013777 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73db795d_bb80_427f_bced_334e30da3621.slice/crio-72e7dd340125ade95b3ca5370a94964e78c799218a3117e7600adf984b006d5c WatchSource:0}: Error finding container 72e7dd340125ade95b3ca5370a94964e78c799218a3117e7600adf984b006d5c: Status 404 returned error can't find the container with id 72e7dd340125ade95b3ca5370a94964e78c799218a3117e7600adf984b006d5c Nov 02 15:15:01.968197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:01.968148 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" event={"ID":"e42e5418-c37e-47cc-83a6-515385294a6f","Type":"ContainerStarted","Data":"772007628c555e46e611041a5ac858cbd8a2c0b198f8ebc150fcec5b773a0633"} Nov 02 15:15:01.969794 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:01.969756 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" event={"ID":"73db795d-bb80-427f-bced-334e30da3621","Type":"ContainerStarted","Data":"72e7dd340125ade95b3ca5370a94964e78c799218a3117e7600adf984b006d5c"} Nov 02 15:15:01.974946 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:01.971170 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" event={"ID":"bf6f0083-090a-45bc-8d6e-b42d7dd1629a","Type":"ContainerStarted","Data":"1fe9392a840a4abbd02f6be481b7e4080c69e1d769cb29091faacf35cf52dbd0"} Nov 02 15:15:06.108128 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.108103 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4"] Nov 02 15:15:06.121435 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.121411 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:06.167802 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.167771 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4"] Nov 02 15:15:06.170603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.170582 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:15:06.292361 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.292294 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt8w7\" (UniqueName: \"kubernetes.io/projected/28fdc0c2-4270-4cb6-9de4-f90544fb9650-kube-api-access-wt8w7\") pod \"maas-api-revoke-sub-09514c5c-r78f4\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:06.292479 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.292462 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09514c5c-r78f4\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:06.292524 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.292503 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09514c5c-r78f4\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:06.393365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.393340 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09514c5c-r78f4\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:06.393461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.393394 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09514c5c-r78f4\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:06.393461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.393448 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt8w7\" (UniqueName: \"kubernetes.io/projected/28fdc0c2-4270-4cb6-9de4-f90544fb9650-kube-api-access-wt8w7\") pod \"maas-api-revoke-sub-09514c5c-r78f4\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:06.394078 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.394058 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09514c5c-r78f4\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:06.395520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.395499 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09514c5c-r78f4\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:06.450445 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.450413 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt8w7\" (UniqueName: \"kubernetes.io/projected/28fdc0c2-4270-4cb6-9de4-f90544fb9650-kube-api-access-wt8w7\") pod \"maas-api-revoke-sub-09514c5c-r78f4\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:06.734146 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:06.734114 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:07.009910 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.009877 5476 generic.go:356] "Generic (PLEG): container finished" podID="e42e5418-c37e-47cc-83a6-515385294a6f" containerID="bb3eaf820b9c06fbb913a2d4a47e6e277b6ae175ac4e4a6fe3a464a3029dc026" exitCode=60 Nov 02 15:15:07.010042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.009957 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" event={"ID":"e42e5418-c37e-47cc-83a6-515385294a6f","Type":"ContainerDied","Data":"bb3eaf820b9c06fbb913a2d4a47e6e277b6ae175ac4e4a6fe3a464a3029dc026"} Nov 02 15:15:07.010216 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.010193 5476 scope.go:122] "RemoveContainer" containerID="bb3eaf820b9c06fbb913a2d4a47e6e277b6ae175ac4e4a6fe3a464a3029dc026" Nov 02 15:15:07.011793 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.011767 5476 generic.go:356] "Generic (PLEG): container finished" podID="73db795d-bb80-427f-bced-334e30da3621" containerID="f20b5afda598b3d687e15ebf331f9b978dc4b6564a5e1edc36bdbe8037da93f3" exitCode=60 Nov 02 15:15:07.011950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.011918 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" event={"ID":"73db795d-bb80-427f-bced-334e30da3621","Type":"ContainerDied","Data":"f20b5afda598b3d687e15ebf331f9b978dc4b6564a5e1edc36bdbe8037da93f3"} Nov 02 15:15:07.012282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.012267 5476 scope.go:122] "RemoveContainer" containerID="f20b5afda598b3d687e15ebf331f9b978dc4b6564a5e1edc36bdbe8037da93f3" Nov 02 15:15:07.014260 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.014198 5476 generic.go:356] "Generic (PLEG): container finished" podID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerID="ea014abd654a08f0c7bd73a83aa0a28e46c9dc5a7981ae883df82ab685402a7b" exitCode=60 Nov 02 15:15:07.014324 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.014273 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" event={"ID":"bf6f0083-090a-45bc-8d6e-b42d7dd1629a","Type":"ContainerDied","Data":"ea014abd654a08f0c7bd73a83aa0a28e46c9dc5a7981ae883df82ab685402a7b"} Nov 02 15:15:07.014536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.014518 5476 scope.go:122] "RemoveContainer" containerID="ea014abd654a08f0c7bd73a83aa0a28e46c9dc5a7981ae883df82ab685402a7b" Nov 02 15:15:07.016136 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.016065 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" event={"ID":"6ba676a7-71bc-4cce-98ea-4bd83a246df7","Type":"ContainerStarted","Data":"50e4de18b9f166111dcbdce4c89002442ce65b6bcb9c325c876f63f07d56d9c7"} Nov 02 15:15:07.105571 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.105545 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4"] Nov 02 15:15:07.174823 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:15:07.174761 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fdc0c2_4270_4cb6_9de4_f90544fb9650.slice/crio-1bbb2a4fb48bc9c0e9bd7eca31db1287bbd6921c54ca3887ea4a1a0a3cc01c41 WatchSource:0}: Error finding container 1bbb2a4fb48bc9c0e9bd7eca31db1287bbd6921c54ca3887ea4a1a0a3cc01c41: Status 404 returned error can't find the container with id 1bbb2a4fb48bc9c0e9bd7eca31db1287bbd6921c54ca3887ea4a1a0a3cc01c41 Nov 02 15:15:07.228105 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:07.228041 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" podStartSLOduration=1.88944315 podStartE2EDuration="7.228022681s" totalImagesPullingTime="5.338579531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:07.225408214 +0000 UTC m=+1009.428286272" watchObservedRunningTime="2026-11-02 15:15:07.228022681 +0000 UTC m=+1009.430900741" Nov 02 15:15:08.023698 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.023667 5476 generic.go:356] "Generic (PLEG): container finished" podID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerID="64c38d18a519ef1df161ec1d5d95116e5906e8d533859ab1cf7ab70f9ba24b30" exitCode=60 Nov 02 15:15:08.023804 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.023750 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" event={"ID":"bf6f0083-090a-45bc-8d6e-b42d7dd1629a","Type":"ContainerDied","Data":"64c38d18a519ef1df161ec1d5d95116e5906e8d533859ab1cf7ab70f9ba24b30"} Nov 02 15:15:08.023860 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.023819 5476 scope.go:122] "RemoveContainer" containerID="ea014abd654a08f0c7bd73a83aa0a28e46c9dc5a7981ae883df82ab685402a7b" Nov 02 15:15:08.024104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.024085 5476 scope.go:122] "RemoveContainer" containerID="64c38d18a519ef1df161ec1d5d95116e5906e8d533859ab1cf7ab70f9ba24b30" Nov 02 15:15:08.024399 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:08.024378 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq_odh-ai-gateway-infra(bf6f0083-090a-45bc-8d6e-b42d7dd1629a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" Nov 02 15:15:08.026187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.026161 5476 generic.go:356] "Generic (PLEG): container finished" podID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerID="50e4de18b9f166111dcbdce4c89002442ce65b6bcb9c325c876f63f07d56d9c7" exitCode=60 Nov 02 15:15:08.026287 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.026202 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" event={"ID":"6ba676a7-71bc-4cce-98ea-4bd83a246df7","Type":"ContainerDied","Data":"50e4de18b9f166111dcbdce4c89002442ce65b6bcb9c325c876f63f07d56d9c7"} Nov 02 15:15:08.026491 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.026474 5476 scope.go:122] "RemoveContainer" containerID="50e4de18b9f166111dcbdce4c89002442ce65b6bcb9c325c876f63f07d56d9c7" Nov 02 15:15:08.028108 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.028084 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" event={"ID":"28fdc0c2-4270-4cb6-9de4-f90544fb9650","Type":"ContainerStarted","Data":"7ed1f1bf8b5f21716b59c7ae4bbd8019c0b8e70a0be9b04f2b29ff8d303c30d9"} Nov 02 15:15:08.028208 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.028149 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" event={"ID":"28fdc0c2-4270-4cb6-9de4-f90544fb9650","Type":"ContainerStarted","Data":"1bbb2a4fb48bc9c0e9bd7eca31db1287bbd6921c54ca3887ea4a1a0a3cc01c41"} Nov 02 15:15:08.031110 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.031082 5476 generic.go:356] "Generic (PLEG): container finished" podID="e42e5418-c37e-47cc-83a6-515385294a6f" containerID="46e5881b1a4b37c05539de3bdd5ddc2052eee76f6bfbd98f8834d5b7e3abbbb5" exitCode=60 Nov 02 15:15:08.031183 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.031160 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" event={"ID":"e42e5418-c37e-47cc-83a6-515385294a6f","Type":"ContainerDied","Data":"46e5881b1a4b37c05539de3bdd5ddc2052eee76f6bfbd98f8834d5b7e3abbbb5"} Nov 02 15:15:08.031402 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.031386 5476 scope.go:122] "RemoveContainer" containerID="46e5881b1a4b37c05539de3bdd5ddc2052eee76f6bfbd98f8834d5b7e3abbbb5" Nov 02 15:15:08.031629 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:08.031605 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q_odh-ai-gateway-infra(e42e5418-c37e-47cc-83a6-515385294a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" Nov 02 15:15:08.033202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.033180 5476 generic.go:356] "Generic (PLEG): container finished" podID="73db795d-bb80-427f-bced-334e30da3621" containerID="bd056eae93b59fa9b4d3713662d376a890dcb280cfe37f2a829ef13e601bdc64" exitCode=60 Nov 02 15:15:08.033285 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.033253 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" event={"ID":"73db795d-bb80-427f-bced-334e30da3621","Type":"ContainerDied","Data":"bd056eae93b59fa9b4d3713662d376a890dcb280cfe37f2a829ef13e601bdc64"} Nov 02 15:15:08.033564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.033550 5476 scope.go:122] "RemoveContainer" containerID="bd056eae93b59fa9b4d3713662d376a890dcb280cfe37f2a829ef13e601bdc64" Nov 02 15:15:08.033787 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:08.033764 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s_odh-ai-gateway-infra(73db795d-bb80-427f-bced-334e30da3621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" podUID="73db795d-bb80-427f-bced-334e30da3621" Nov 02 15:15:08.092349 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.092331 5476 scope.go:122] "RemoveContainer" containerID="bb3eaf820b9c06fbb913a2d4a47e6e277b6ae175ac4e4a6fe3a464a3029dc026" Nov 02 15:15:08.108650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.108630 5476 scope.go:122] "RemoveContainer" containerID="f20b5afda598b3d687e15ebf331f9b978dc4b6564a5e1edc36bdbe8037da93f3" Nov 02 15:15:08.352335 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:08.352269 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" podStartSLOduration=2.3522513099999998 podStartE2EDuration="2.35225131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:08.350867468 +0000 UTC m=+1010.553745530" watchObservedRunningTime="2026-11-02 15:15:08.35225131 +0000 UTC m=+1010.555129371" Nov 02 15:15:09.041774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:09.041741 5476 scope.go:122] "RemoveContainer" containerID="bd056eae93b59fa9b4d3713662d376a890dcb280cfe37f2a829ef13e601bdc64" Nov 02 15:15:09.042091 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:09.042059 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s_odh-ai-gateway-infra(73db795d-bb80-427f-bced-334e30da3621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" podUID="73db795d-bb80-427f-bced-334e30da3621" Nov 02 15:15:09.043469 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:09.043448 5476 scope.go:122] "RemoveContainer" containerID="64c38d18a519ef1df161ec1d5d95116e5906e8d533859ab1cf7ab70f9ba24b30" Nov 02 15:15:09.043730 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:09.043704 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq_odh-ai-gateway-infra(bf6f0083-090a-45bc-8d6e-b42d7dd1629a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" Nov 02 15:15:09.044932 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:09.044910 5476 generic.go:356] "Generic (PLEG): container finished" podID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerID="f32273ef7df31fd2c67ea13ab88f69aa791b07e2c9771037e152d990c76d9f63" exitCode=60 Nov 02 15:15:09.045061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:09.045015 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" event={"ID":"6ba676a7-71bc-4cce-98ea-4bd83a246df7","Type":"ContainerDied","Data":"f32273ef7df31fd2c67ea13ab88f69aa791b07e2c9771037e152d990c76d9f63"} Nov 02 15:15:09.045157 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:09.045117 5476 scope.go:122] "RemoveContainer" containerID="50e4de18b9f166111dcbdce4c89002442ce65b6bcb9c325c876f63f07d56d9c7" Nov 02 15:15:09.045229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:09.045211 5476 scope.go:122] "RemoveContainer" containerID="f32273ef7df31fd2c67ea13ab88f69aa791b07e2c9771037e152d990c76d9f63" Nov 02 15:15:09.045462 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:09.045439 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2_odh-ai-gateway-infra(6ba676a7-71bc-4cce-98ea-4bd83a246df7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" Nov 02 15:15:09.046616 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:09.046594 5476 generic.go:356] "Generic (PLEG): container finished" podID="28fdc0c2-4270-4cb6-9de4-f90544fb9650" containerID="7ed1f1bf8b5f21716b59c7ae4bbd8019c0b8e70a0be9b04f2b29ff8d303c30d9" exitCode=0 Nov 02 15:15:09.046687 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:09.046670 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" event={"ID":"28fdc0c2-4270-4cb6-9de4-f90544fb9650","Type":"ContainerDied","Data":"7ed1f1bf8b5f21716b59c7ae4bbd8019c0b8e70a0be9b04f2b29ff8d303c30d9"} Nov 02 15:15:09.048408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:09.048387 5476 scope.go:122] "RemoveContainer" containerID="46e5881b1a4b37c05539de3bdd5ddc2052eee76f6bfbd98f8834d5b7e3abbbb5" Nov 02 15:15:09.048585 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:09.048570 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q_odh-ai-gateway-infra(e42e5418-c37e-47cc-83a6-515385294a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" Nov 02 15:15:10.056064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:10.056033 5476 scope.go:122] "RemoveContainer" containerID="f32273ef7df31fd2c67ea13ab88f69aa791b07e2c9771037e152d990c76d9f63" Nov 02 15:15:10.056453 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:10.056306 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2_odh-ai-gateway-infra(6ba676a7-71bc-4cce-98ea-4bd83a246df7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" Nov 02 15:15:10.196608 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:10.196577 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:10.235509 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:10.235477 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-xks-ca\") pod \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " Nov 02 15:15:10.235643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:10.235539 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-openshift-ca\") pod \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " Nov 02 15:15:10.235869 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:10.235846 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-openshift-ca" pod "28fdc0c2-4270-4cb6-9de4-f90544fb9650" (UID: "28fdc0c2-4270-4cb6-9de4-f90544fb9650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:10.236151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:10.236132 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:10.237684 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:10.237661 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-xks-ca" pod "28fdc0c2-4270-4cb6-9de4-f90544fb9650" (UID: "28fdc0c2-4270-4cb6-9de4-f90544fb9650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:15:10.336516 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:10.336432 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fdc0c2-4270-4cb6-9de4-f90544fb9650-kube-api-access-wt8w7\" (UniqueName: \"kubernetes.io/projected/28fdc0c2-4270-4cb6-9de4-f90544fb9650-kube-api-access-wt8w7\") pod \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\" (UID: \"28fdc0c2-4270-4cb6-9de4-f90544fb9650\") " Nov 02 15:15:10.336876 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:10.336860 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28fdc0c2-4270-4cb6-9de4-f90544fb9650-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:10.338584 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:10.338551 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fdc0c2-4270-4cb6-9de4-f90544fb9650-kube-api-access-wt8w7" pod "28fdc0c2-4270-4cb6-9de4-f90544fb9650" (UID: "28fdc0c2-4270-4cb6-9de4-f90544fb9650"). InnerVolumeSpecName "kube-api-access-wt8w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:10.437747 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:10.437716 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt8w7\" (UniqueName: \"kubernetes.io/projected/28fdc0c2-4270-4cb6-9de4-f90544fb9650-kube-api-access-wt8w7\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:11.062971 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:11.062938 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" Nov 02 15:15:11.063388 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:11.062940 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4" event={"ID":"28fdc0c2-4270-4cb6-9de4-f90544fb9650","Type":"ContainerDied","Data":"1bbb2a4fb48bc9c0e9bd7eca31db1287bbd6921c54ca3887ea4a1a0a3cc01c41"} Nov 02 15:15:11.063388 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:11.063052 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bbb2a4fb48bc9c0e9bd7eca31db1287bbd6921c54ca3887ea4a1a0a3cc01c41" Nov 02 15:15:12.277438 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:12.277405 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:15:20.145612 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:20.145576 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:15:20.915531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:20.915501 5476 scope.go:122] "RemoveContainer" containerID="64c38d18a519ef1df161ec1d5d95116e5906e8d533859ab1cf7ab70f9ba24b30" Nov 02 15:15:20.915744 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:20.915698 5476 scope.go:122] "RemoveContainer" containerID="f32273ef7df31fd2c67ea13ab88f69aa791b07e2c9771037e152d990c76d9f63" Nov 02 15:15:21.916073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:21.916045 5476 scope.go:122] "RemoveContainer" containerID="46e5881b1a4b37c05539de3bdd5ddc2052eee76f6bfbd98f8834d5b7e3abbbb5" Nov 02 15:15:21.916391 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:21.916213 5476 scope.go:122] "RemoveContainer" containerID="bd056eae93b59fa9b4d3713662d376a890dcb280cfe37f2a829ef13e601bdc64" Nov 02 15:15:22.130907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:22.130879 5476 generic.go:356] "Generic (PLEG): container finished" podID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerID="4270fa750329ff331a6f3061235805887a13112ca0682bb429e156417dfa6f73" exitCode=60 Nov 02 15:15:22.131071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:22.130954 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" event={"ID":"bf6f0083-090a-45bc-8d6e-b42d7dd1629a","Type":"ContainerDied","Data":"4270fa750329ff331a6f3061235805887a13112ca0682bb429e156417dfa6f73"} Nov 02 15:15:22.131071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:22.131039 5476 scope.go:122] "RemoveContainer" containerID="64c38d18a519ef1df161ec1d5d95116e5906e8d533859ab1cf7ab70f9ba24b30" Nov 02 15:15:22.131397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:22.131373 5476 scope.go:122] "RemoveContainer" containerID="4270fa750329ff331a6f3061235805887a13112ca0682bb429e156417dfa6f73" Nov 02 15:15:22.131670 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:22.131650 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq_odh-ai-gateway-infra(bf6f0083-090a-45bc-8d6e-b42d7dd1629a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" Nov 02 15:15:22.132968 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:22.132949 5476 generic.go:356] "Generic (PLEG): container finished" podID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerID="bcbb308f531062ee5092cf24702bed388f7cb1480ab8f4780e237d1af99e0626" exitCode=60 Nov 02 15:15:22.133088 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:22.133035 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" event={"ID":"6ba676a7-71bc-4cce-98ea-4bd83a246df7","Type":"ContainerDied","Data":"bcbb308f531062ee5092cf24702bed388f7cb1480ab8f4780e237d1af99e0626"} Nov 02 15:15:22.133327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:22.133314 5476 scope.go:122] "RemoveContainer" containerID="bcbb308f531062ee5092cf24702bed388f7cb1480ab8f4780e237d1af99e0626" Nov 02 15:15:22.133531 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:22.133517 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2_odh-ai-gateway-infra(6ba676a7-71bc-4cce-98ea-4bd83a246df7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" Nov 02 15:15:22.145052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:22.145034 5476 scope.go:122] "RemoveContainer" containerID="f32273ef7df31fd2c67ea13ab88f69aa791b07e2c9771037e152d990c76d9f63" Nov 02 15:15:23.142388 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.142356 5476 generic.go:356] "Generic (PLEG): container finished" podID="e42e5418-c37e-47cc-83a6-515385294a6f" containerID="7ee112b3155d807cbf535bbf0940d5afd855f527735ea20a578ece3ab4ded1ab" exitCode=60 Nov 02 15:15:23.142879 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.142427 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" event={"ID":"e42e5418-c37e-47cc-83a6-515385294a6f","Type":"ContainerDied","Data":"7ee112b3155d807cbf535bbf0940d5afd855f527735ea20a578ece3ab4ded1ab"} Nov 02 15:15:23.142879 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.142491 5476 scope.go:122] "RemoveContainer" containerID="46e5881b1a4b37c05539de3bdd5ddc2052eee76f6bfbd98f8834d5b7e3abbbb5" Nov 02 15:15:23.143028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.142903 5476 scope.go:122] "RemoveContainer" containerID="7ee112b3155d807cbf535bbf0940d5afd855f527735ea20a578ece3ab4ded1ab" Nov 02 15:15:23.143301 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:23.143267 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q_odh-ai-gateway-infra(e42e5418-c37e-47cc-83a6-515385294a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" Nov 02 15:15:23.144859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.144840 5476 generic.go:356] "Generic (PLEG): container finished" podID="73db795d-bb80-427f-bced-334e30da3621" containerID="54448d926c20de6d1170339905411c599603302b65b2b257038316b99c852ffc" exitCode=60 Nov 02 15:15:23.144967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.144914 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" event={"ID":"73db795d-bb80-427f-bced-334e30da3621","Type":"ContainerDied","Data":"54448d926c20de6d1170339905411c599603302b65b2b257038316b99c852ffc"} Nov 02 15:15:23.145276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.145260 5476 scope.go:122] "RemoveContainer" containerID="54448d926c20de6d1170339905411c599603302b65b2b257038316b99c852ffc" Nov 02 15:15:23.145523 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:23.145501 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s_odh-ai-gateway-infra(73db795d-bb80-427f-bced-334e30da3621)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" podUID="73db795d-bb80-427f-bced-334e30da3621" Nov 02 15:15:23.155852 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.155832 5476 scope.go:122] "RemoveContainer" containerID="bd056eae93b59fa9b4d3713662d376a890dcb280cfe37f2a829ef13e601bdc64" Nov 02 15:15:23.254199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.254167 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq"] Nov 02 15:15:23.369371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.369337 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2"] Nov 02 15:15:23.412288 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.412224 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" Nov 02 15:15:23.480027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.476677 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-kube-api-access-4sbsv\" (UniqueName: \"kubernetes.io/projected/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-kube-api-access-4sbsv\") pod \"bf6f0083-090a-45bc-8d6e-b42d7dd1629a\" (UID: \"bf6f0083-090a-45bc-8d6e-b42d7dd1629a\") " Nov 02 15:15:23.480027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.476823 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-maas-api-service-ca\") pod \"bf6f0083-090a-45bc-8d6e-b42d7dd1629a\" (UID: \"bf6f0083-090a-45bc-8d6e-b42d7dd1629a\") " Nov 02 15:15:23.480027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.477633 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-maas-api-service-ca" pod "bf6f0083-090a-45bc-8d6e-b42d7dd1629a" (UID: "bf6f0083-090a-45bc-8d6e-b42d7dd1629a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:23.490769 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.490730 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-kube-api-access-4sbsv" pod "bf6f0083-090a-45bc-8d6e-b42d7dd1629a" (UID: "bf6f0083-090a-45bc-8d6e-b42d7dd1629a"). InnerVolumeSpecName "kube-api-access-4sbsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:23.547561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.547533 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" Nov 02 15:15:23.578305 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.578276 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:23.578305 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.578301 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sbsv\" (UniqueName: \"kubernetes.io/projected/bf6f0083-090a-45bc-8d6e-b42d7dd1629a-kube-api-access-4sbsv\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:23.678869 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.678791 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba676a7-71bc-4cce-98ea-4bd83a246df7-kube-api-access-5x4wt\" (UniqueName: \"kubernetes.io/projected/6ba676a7-71bc-4cce-98ea-4bd83a246df7-kube-api-access-5x4wt\") pod \"6ba676a7-71bc-4cce-98ea-4bd83a246df7\" (UID: \"6ba676a7-71bc-4cce-98ea-4bd83a246df7\") " Nov 02 15:15:23.679016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.678916 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba676a7-71bc-4cce-98ea-4bd83a246df7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba676a7-71bc-4cce-98ea-4bd83a246df7-maas-api-service-ca\") pod \"6ba676a7-71bc-4cce-98ea-4bd83a246df7\" (UID: \"6ba676a7-71bc-4cce-98ea-4bd83a246df7\") " Nov 02 15:15:23.679312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.679285 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba676a7-71bc-4cce-98ea-4bd83a246df7-maas-api-service-ca" pod "6ba676a7-71bc-4cce-98ea-4bd83a246df7" (UID: "6ba676a7-71bc-4cce-98ea-4bd83a246df7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:23.681186 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.681061 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba676a7-71bc-4cce-98ea-4bd83a246df7-kube-api-access-5x4wt" pod "6ba676a7-71bc-4cce-98ea-4bd83a246df7" (UID: "6ba676a7-71bc-4cce-98ea-4bd83a246df7"). InnerVolumeSpecName "kube-api-access-5x4wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:23.780230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.780180 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x4wt\" (UniqueName: \"kubernetes.io/projected/6ba676a7-71bc-4cce-98ea-4bd83a246df7-kube-api-access-5x4wt\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:23.780230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:23.780221 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba676a7-71bc-4cce-98ea-4bd83a246df7-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:24.157377 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.157338 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" event={"ID":"bf6f0083-090a-45bc-8d6e-b42d7dd1629a","Type":"ContainerDied","Data":"1fe9392a840a4abbd02f6be481b7e4080c69e1d769cb29091faacf35cf52dbd0"} Nov 02 15:15:24.157908 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.157403 5476 scope.go:122] "RemoveContainer" containerID="4270fa750329ff331a6f3061235805887a13112ca0682bb429e156417dfa6f73" Nov 02 15:15:24.157908 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.157350 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq" Nov 02 15:15:24.159197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.159177 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" Nov 02 15:15:24.159325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.159273 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2" event={"ID":"6ba676a7-71bc-4cce-98ea-4bd83a246df7","Type":"ContainerDied","Data":"09fe224d99be4109de4db2aa5c701537e08ad85577a2e7b5bdf89c4ac98ba218"} Nov 02 15:15:24.169506 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.169482 5476 scope.go:122] "RemoveContainer" containerID="bcbb308f531062ee5092cf24702bed388f7cb1480ab8f4780e237d1af99e0626" Nov 02 15:15:24.266464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.266368 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q"] Nov 02 15:15:24.321954 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.321880 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2"] Nov 02 15:15:24.336232 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.336177 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893875-h4db2"] Nov 02 15:15:24.511537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.511065 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s"] Nov 02 15:15:24.532324 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.520453 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq"] Nov 02 15:15:24.604032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.596066 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893875-vc5rq"] Nov 02 15:15:24.627032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.618362 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" Nov 02 15:15:24.688017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.687232 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e42e5418-c37e-47cc-83a6-515385294a6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42e5418-c37e-47cc-83a6-515385294a6f-maas-api-service-ca\") pod \"e42e5418-c37e-47cc-83a6-515385294a6f\" (UID: \"e42e5418-c37e-47cc-83a6-515385294a6f\") " Nov 02 15:15:24.688017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.687324 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e42e5418-c37e-47cc-83a6-515385294a6f-kube-api-access-fp7ks\" (UniqueName: \"kubernetes.io/projected/e42e5418-c37e-47cc-83a6-515385294a6f-kube-api-access-fp7ks\") pod \"e42e5418-c37e-47cc-83a6-515385294a6f\" (UID: \"e42e5418-c37e-47cc-83a6-515385294a6f\") " Nov 02 15:15:24.692017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.688420 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e42e5418-c37e-47cc-83a6-515385294a6f-maas-api-service-ca" pod "e42e5418-c37e-47cc-83a6-515385294a6f" (UID: "e42e5418-c37e-47cc-83a6-515385294a6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:24.727040 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.726658 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42e5418-c37e-47cc-83a6-515385294a6f-kube-api-access-fp7ks" pod "e42e5418-c37e-47cc-83a6-515385294a6f" (UID: "e42e5418-c37e-47cc-83a6-515385294a6f"). InnerVolumeSpecName "kube-api-access-fp7ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:24.781638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.781607 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" Nov 02 15:15:24.788552 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.788484 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73db795d-bb80-427f-bced-334e30da3621-kube-api-access-cmb9l\" (UniqueName: \"kubernetes.io/projected/73db795d-bb80-427f-bced-334e30da3621-kube-api-access-cmb9l\") pod \"73db795d-bb80-427f-bced-334e30da3621\" (UID: \"73db795d-bb80-427f-bced-334e30da3621\") " Nov 02 15:15:24.788552 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.788548 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73db795d-bb80-427f-bced-334e30da3621-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73db795d-bb80-427f-bced-334e30da3621-maas-api-service-ca\") pod \"73db795d-bb80-427f-bced-334e30da3621\" (UID: \"73db795d-bb80-427f-bced-334e30da3621\") " Nov 02 15:15:24.788793 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.788778 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e42e5418-c37e-47cc-83a6-515385294a6f-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:24.788865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.788793 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp7ks\" (UniqueName: \"kubernetes.io/projected/e42e5418-c37e-47cc-83a6-515385294a6f-kube-api-access-fp7ks\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:24.788918 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.788874 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73db795d-bb80-427f-bced-334e30da3621-maas-api-service-ca" pod "73db795d-bb80-427f-bced-334e30da3621" (UID: "73db795d-bb80-427f-bced-334e30da3621"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:24.790632 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.790600 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73db795d-bb80-427f-bced-334e30da3621-kube-api-access-cmb9l" pod "73db795d-bb80-427f-bced-334e30da3621" (UID: "73db795d-bb80-427f-bced-334e30da3621"). InnerVolumeSpecName "kube-api-access-cmb9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:24.865514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.864706 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v"] Nov 02 15:15:24.865514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865469 5476 cpu_manager.go:414] "containerMap: removing container" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerName="cleanup" Nov 02 15:15:24.865514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865490 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerName="cleanup" Nov 02 15:15:24.865514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865503 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerName="cleanup" Nov 02 15:15:24.865514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865512 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerName="cleanup" Nov 02 15:15:24.865514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865523 5476 cpu_manager.go:414] "containerMap: removing container" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865534 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865557 5476 cpu_manager.go:414] "containerMap: removing container" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865565 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865579 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865586 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865611 5476 cpu_manager.go:414] "containerMap: removing container" podUID="73db795d-bb80-427f-bced-334e30da3621" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865619 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73db795d-bb80-427f-bced-334e30da3621" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865642 5476 cpu_manager.go:414] "containerMap: removing container" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865649 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865662 5476 cpu_manager.go:414] "containerMap: removing container" podUID="28fdc0c2-4270-4cb6-9de4-f90544fb9650" containerName="revoke-keys" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865670 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fdc0c2-4270-4cb6-9de4-f90544fb9650" containerName="revoke-keys" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865685 5476 cpu_manager.go:414] "containerMap: removing container" podUID="73db795d-bb80-427f-bced-334e30da3621" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865693 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73db795d-bb80-427f-bced-334e30da3621" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865804 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865818 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="73db795d-bb80-427f-bced-334e30da3621" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865831 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865841 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865852 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fdc0c2-4270-4cb6-9de4-f90544fb9650" containerName="revoke-keys" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865867 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="73db795d-bb80-427f-bced-334e30da3621" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865876 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865887 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" containerName="cleanup" Nov 02 15:15:24.865889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865899 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" containerName="cleanup" Nov 02 15:15:24.866594 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.865909 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerName="cleanup" Nov 02 15:15:24.869801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.869778 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.889456 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.889422 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-dshm\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.889628 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.889468 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tls-certs\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.889628 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.889542 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tmp-dir\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.889628 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.889618 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-model-cache\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.889777 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.889666 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qsvc\" (UniqueName: \"kubernetes.io/projected/00be7b9f-4f51-4c7e-b795-76e23f06adeb-kube-api-access-2qsvc\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.889777 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.889708 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-home\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.889873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.889778 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmb9l\" (UniqueName: \"kubernetes.io/projected/73db795d-bb80-427f-bced-334e30da3621-kube-api-access-cmb9l\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:24.889873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.889788 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73db795d-bb80-427f-bced-334e30da3621-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:24.927650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.925630 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-624908\"/\"premium-w1-624908-kserve-self-signed-certs\"" Nov 02 15:15:24.934418 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.934324 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v"] Nov 02 15:15:24.991337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.991297 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tmp-dir\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.991514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.991424 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-model-cache\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.991514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.991459 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qsvc\" (UniqueName: \"kubernetes.io/projected/00be7b9f-4f51-4c7e-b795-76e23f06adeb-kube-api-access-2qsvc\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.991592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.991520 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-home\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.991635 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.991603 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-dshm\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.991669 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.991649 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tls-certs\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.991808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.991784 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tmp-dir\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.991852 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.991800 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-model-cache\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.992074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.992054 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-home\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.993980 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.993954 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-dshm\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:24.994344 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:24.994325 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tls-certs\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:25.053590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.053507 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qsvc\" (UniqueName: \"kubernetes.io/projected/00be7b9f-4f51-4c7e-b795-76e23f06adeb-kube-api-access-2qsvc\") pod \"premium-w1-624908-kserve-679ccd8856-l522v\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:25.172978 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.172947 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" Nov 02 15:15:25.172978 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.172968 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q" event={"ID":"e42e5418-c37e-47cc-83a6-515385294a6f","Type":"ContainerDied","Data":"772007628c555e46e611041a5ac858cbd8a2c0b198f8ebc150fcec5b773a0633"} Nov 02 15:15:25.173462 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.173057 5476 scope.go:122] "RemoveContainer" containerID="7ee112b3155d807cbf535bbf0940d5afd855f527735ea20a578ece3ab4ded1ab" Nov 02 15:15:25.174358 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.174339 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" event={"ID":"73db795d-bb80-427f-bced-334e30da3621","Type":"ContainerDied","Data":"72e7dd340125ade95b3ca5370a94964e78c799218a3117e7600adf984b006d5c"} Nov 02 15:15:25.174419 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.174404 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s" Nov 02 15:15:25.184703 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.184613 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:25.186265 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.186236 5476 scope.go:122] "RemoveContainer" containerID="54448d926c20de6d1170339905411c599603302b65b2b257038316b99c852ffc" Nov 02 15:15:25.330473 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.330392 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s"] Nov 02 15:15:25.335419 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.335387 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893875-85p5s"] Nov 02 15:15:25.440056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.440024 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q"] Nov 02 15:15:25.444928 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.444899 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893875-wfm7q"] Nov 02 15:15:25.582895 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.582822 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v"] Nov 02 15:15:25.588354 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:15:25.588322 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00be7b9f_4f51_4c7e_b795_76e23f06adeb.slice/crio-aaf1e5e368eee45b40c92787d413f0167f96f001438e8517cf0835fe5c7e4a7d WatchSource:0}: Error finding container aaf1e5e368eee45b40c92787d413f0167f96f001438e8517cf0835fe5c7e4a7d: Status 404 returned error can't find the container with id aaf1e5e368eee45b40c92787d413f0167f96f001438e8517cf0835fe5c7e4a7d Nov 02 15:15:25.921722 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.921688 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" path="/var/lib/kubelet/pods/6ba676a7-71bc-4cce-98ea-4bd83a246df7/volumes" Nov 02 15:15:25.922110 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.922094 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73db795d-bb80-427f-bced-334e30da3621" path="/var/lib/kubelet/pods/73db795d-bb80-427f-bced-334e30da3621/volumes" Nov 02 15:15:25.922439 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.922427 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" path="/var/lib/kubelet/pods/bf6f0083-090a-45bc-8d6e-b42d7dd1629a/volumes" Nov 02 15:15:25.923067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:25.923052 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" path="/var/lib/kubelet/pods/e42e5418-c37e-47cc-83a6-515385294a6f/volumes" Nov 02 15:15:26.183219 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:26.183129 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" event={"ID":"00be7b9f-4f51-4c7e-b795-76e23f06adeb","Type":"ContainerStarted","Data":"6f2b8afc1145f5b43b5a0ab82d55360a71d9f9ce93380a6ebfffa420b96974b8"} Nov 02 15:15:26.183219 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:26.183180 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" event={"ID":"00be7b9f-4f51-4c7e-b795-76e23f06adeb","Type":"ContainerStarted","Data":"aaf1e5e368eee45b40c92787d413f0167f96f001438e8517cf0835fe5c7e4a7d"} Nov 02 15:15:26.184695 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:26.184672 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:26.299483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:26.299416 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podStartSLOduration=2.299401668 podStartE2EDuration="2.299401668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:26.298484405 +0000 UTC m=+1028.501362476" watchObservedRunningTime="2026-11-02 15:15:26.299401668 +0000 UTC m=+1028.502279731" Nov 02 15:15:28.651245 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651209 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp"] Nov 02 15:15:28.651706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651692 5476 cpu_manager.go:414] "containerMap: removing container" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" containerName="cleanup" Nov 02 15:15:28.651745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651706 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e42e5418-c37e-47cc-83a6-515385294a6f" containerName="cleanup" Nov 02 15:15:28.651745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651727 5476 cpu_manager.go:414] "containerMap: removing container" podUID="73db795d-bb80-427f-bced-334e30da3621" containerName="cleanup" Nov 02 15:15:28.651745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651732 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73db795d-bb80-427f-bced-334e30da3621" containerName="cleanup" Nov 02 15:15:28.651844 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651760 5476 cpu_manager.go:414] "containerMap: removing container" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerName="cleanup" Nov 02 15:15:28.651844 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651765 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerName="cleanup" Nov 02 15:15:28.651844 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651772 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerName="cleanup" Nov 02 15:15:28.651844 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651776 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerName="cleanup" Nov 02 15:15:28.651952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651846 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba676a7-71bc-4cce-98ea-4bd83a246df7" containerName="cleanup" Nov 02 15:15:28.651952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651857 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6f0083-090a-45bc-8d6e-b42d7dd1629a" containerName="cleanup" Nov 02 15:15:28.651952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.651863 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="73db795d-bb80-427f-bced-334e30da3621" containerName="cleanup" Nov 02 15:15:28.659607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.659586 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:28.711272 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.710972 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp"] Nov 02 15:15:28.733956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.733931 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40383841-8pgxp\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:28.734146 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.733963 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8wds\" (UniqueName: \"kubernetes.io/projected/30682f86-a130-4716-bbf3-6fc4756c0798-kube-api-access-x8wds\") pod \"maas-api-revoke-sub-40383841-8pgxp\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:28.734146 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.734129 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40383841-8pgxp\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:28.754342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.754318 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:15:28.754432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.754360 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:15:28.835200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.835165 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8wds\" (UniqueName: \"kubernetes.io/projected/30682f86-a130-4716-bbf3-6fc4756c0798-kube-api-access-x8wds\") pod \"maas-api-revoke-sub-40383841-8pgxp\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:28.835371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.835261 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40383841-8pgxp\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:28.835426 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.835373 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40383841-8pgxp\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:28.835868 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.835844 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40383841-8pgxp\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:28.837678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.837655 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40383841-8pgxp\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:28.892747 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.892717 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8wds\" (UniqueName: \"kubernetes.io/projected/30682f86-a130-4716-bbf3-6fc4756c0798-kube-api-access-x8wds\") pod \"maas-api-revoke-sub-40383841-8pgxp\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:28.972787 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:28.972707 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:29.346795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:29.346710 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp"] Nov 02 15:15:29.352283 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:15:29.352252 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30682f86_a130_4716_bbf3_6fc4756c0798.slice/crio-2e37219103c84e88d4c9c2b5a605b939811afabcbe80682a67692f3ed1257312 WatchSource:0}: Error finding container 2e37219103c84e88d4c9c2b5a605b939811afabcbe80682a67692f3ed1257312: Status 404 returned error can't find the container with id 2e37219103c84e88d4c9c2b5a605b939811afabcbe80682a67692f3ed1257312 Nov 02 15:15:30.210615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:30.210584 5476 generic.go:356] "Generic (PLEG): container finished" podID="30682f86-a130-4716-bbf3-6fc4756c0798" containerID="040ff0dbb31d484299bf9ec79d70f59fd2060c81ac0607144589a1699963953f" exitCode=0 Nov 02 15:15:30.211052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:30.210644 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" event={"ID":"30682f86-a130-4716-bbf3-6fc4756c0798","Type":"ContainerDied","Data":"040ff0dbb31d484299bf9ec79d70f59fd2060c81ac0607144589a1699963953f"} Nov 02 15:15:30.211052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:30.210709 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" event={"ID":"30682f86-a130-4716-bbf3-6fc4756c0798","Type":"ContainerStarted","Data":"2e37219103c84e88d4c9c2b5a605b939811afabcbe80682a67692f3ed1257312"} Nov 02 15:15:31.357076 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:31.357052 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:31.461492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:31.461469 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-openshift-ca\") pod \"30682f86-a130-4716-bbf3-6fc4756c0798\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " Nov 02 15:15:31.461624 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:31.461513 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30682f86-a130-4716-bbf3-6fc4756c0798-kube-api-access-x8wds\" (UniqueName: \"kubernetes.io/projected/30682f86-a130-4716-bbf3-6fc4756c0798-kube-api-access-x8wds\") pod \"30682f86-a130-4716-bbf3-6fc4756c0798\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " Nov 02 15:15:31.461624 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:31.461609 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-xks-ca\") pod \"30682f86-a130-4716-bbf3-6fc4756c0798\" (UID: \"30682f86-a130-4716-bbf3-6fc4756c0798\") " Nov 02 15:15:31.461853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:31.461826 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-openshift-ca" pod "30682f86-a130-4716-bbf3-6fc4756c0798" (UID: "30682f86-a130-4716-bbf3-6fc4756c0798"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:31.463577 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:31.463552 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-xks-ca" pod "30682f86-a130-4716-bbf3-6fc4756c0798" (UID: "30682f86-a130-4716-bbf3-6fc4756c0798"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:15:31.463675 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:31.463648 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30682f86-a130-4716-bbf3-6fc4756c0798-kube-api-access-x8wds" pod "30682f86-a130-4716-bbf3-6fc4756c0798" (UID: "30682f86-a130-4716-bbf3-6fc4756c0798"). InnerVolumeSpecName "kube-api-access-x8wds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:31.563416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:31.563387 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8wds\" (UniqueName: \"kubernetes.io/projected/30682f86-a130-4716-bbf3-6fc4756c0798-kube-api-access-x8wds\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:31.563416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:31.563414 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:31.563416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:31.563423 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30682f86-a130-4716-bbf3-6fc4756c0798-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:32.223450 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:32.223368 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" event={"ID":"30682f86-a130-4716-bbf3-6fc4756c0798","Type":"ContainerDied","Data":"2e37219103c84e88d4c9c2b5a605b939811afabcbe80682a67692f3ed1257312"} Nov 02 15:15:32.223450 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:32.223401 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp" Nov 02 15:15:32.223450 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:32.223421 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e37219103c84e88d4c9c2b5a605b939811afabcbe80682a67692f3ed1257312" Nov 02 15:15:34.677621 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:34.677586 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:15:35.185217 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:35.185179 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:35.198254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:35.198226 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:35.252402 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:35.252367 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:15:44.403406 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.403372 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g"] Nov 02 15:15:44.403865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.403849 5476 cpu_manager.go:414] "containerMap: removing container" podUID="30682f86-a130-4716-bbf3-6fc4756c0798" containerName="revoke-keys" Nov 02 15:15:44.403966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.403866 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30682f86-a130-4716-bbf3-6fc4756c0798" containerName="revoke-keys" Nov 02 15:15:44.404061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.403974 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="30682f86-a130-4716-bbf3-6fc4756c0798" containerName="revoke-keys" Nov 02 15:15:44.407382 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.407364 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:44.456440 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.456410 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:15:44.502451 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.502418 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:15:44.505029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.504979 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c8200f-x6h4g\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:44.505202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.505138 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c8200f-x6h4g\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:44.505259 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.505240 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j29b\" (UniqueName: \"kubernetes.io/projected/042e5e11-0753-4d6d-b606-e0a2a32db585-kube-api-access-2j29b\") pod \"maas-api-revoke-sub-92c8200f-x6h4g\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:44.505556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.505539 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g"] Nov 02 15:15:44.606792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.606750 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c8200f-x6h4g\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:44.606981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.606816 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c8200f-x6h4g\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:44.606981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.606855 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j29b\" (UniqueName: \"kubernetes.io/projected/042e5e11-0753-4d6d-b606-e0a2a32db585-kube-api-access-2j29b\") pod \"maas-api-revoke-sub-92c8200f-x6h4g\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:44.607424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.607400 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92c8200f-x6h4g\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:44.609327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.609303 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92c8200f-x6h4g\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:44.665535 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.665459 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j29b\" (UniqueName: \"kubernetes.io/projected/042e5e11-0753-4d6d-b606-e0a2a32db585-kube-api-access-2j29b\") pod \"maas-api-revoke-sub-92c8200f-x6h4g\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:44.720485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:44.720448 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:45.093018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:45.092917 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g"] Nov 02 15:15:45.098569 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:15:45.098539 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042e5e11_0753_4d6d_b606_e0a2a32db585.slice/crio-4a03115aa6979e9f2464f8a71b2a9a6bcfb15935598e2f3d16e65a138db9e8c3 WatchSource:0}: Error finding container 4a03115aa6979e9f2464f8a71b2a9a6bcfb15935598e2f3d16e65a138db9e8c3: Status 404 returned error can't find the container with id 4a03115aa6979e9f2464f8a71b2a9a6bcfb15935598e2f3d16e65a138db9e8c3 Nov 02 15:15:45.304762 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:45.304724 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" event={"ID":"042e5e11-0753-4d6d-b606-e0a2a32db585","Type":"ContainerStarted","Data":"4a03115aa6979e9f2464f8a71b2a9a6bcfb15935598e2f3d16e65a138db9e8c3"} Nov 02 15:15:46.312566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:46.312486 5476 generic.go:356] "Generic (PLEG): container finished" podID="042e5e11-0753-4d6d-b606-e0a2a32db585" containerID="ce918191603446d8fea4a965b0b4185cf7767a6c9d4cf25bcfdee97ee565ffb0" exitCode=0 Nov 02 15:15:46.312566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:46.312522 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" event={"ID":"042e5e11-0753-4d6d-b606-e0a2a32db585","Type":"ContainerDied","Data":"ce918191603446d8fea4a965b0b4185cf7767a6c9d4cf25bcfdee97ee565ffb0"} Nov 02 15:15:47.461077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:47.461052 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:47.637553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:47.637518 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-xks-ca\") pod \"042e5e11-0753-4d6d-b606-e0a2a32db585\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " Nov 02 15:15:47.637768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:47.637591 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042e5e11-0753-4d6d-b606-e0a2a32db585-kube-api-access-2j29b\" (UniqueName: \"kubernetes.io/projected/042e5e11-0753-4d6d-b606-e0a2a32db585-kube-api-access-2j29b\") pod \"042e5e11-0753-4d6d-b606-e0a2a32db585\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " Nov 02 15:15:47.637768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:47.637627 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-openshift-ca\") pod \"042e5e11-0753-4d6d-b606-e0a2a32db585\" (UID: \"042e5e11-0753-4d6d-b606-e0a2a32db585\") " Nov 02 15:15:47.638066 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:47.638040 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-openshift-ca" pod "042e5e11-0753-4d6d-b606-e0a2a32db585" (UID: "042e5e11-0753-4d6d-b606-e0a2a32db585"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:47.639742 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:47.639706 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-xks-ca" pod "042e5e11-0753-4d6d-b606-e0a2a32db585" (UID: "042e5e11-0753-4d6d-b606-e0a2a32db585"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:15:47.639877 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:47.639855 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042e5e11-0753-4d6d-b606-e0a2a32db585-kube-api-access-2j29b" pod "042e5e11-0753-4d6d-b606-e0a2a32db585" (UID: "042e5e11-0753-4d6d-b606-e0a2a32db585"). InnerVolumeSpecName "kube-api-access-2j29b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:47.739279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:47.739241 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:47.739279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:47.739273 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j29b\" (UniqueName: \"kubernetes.io/projected/042e5e11-0753-4d6d-b606-e0a2a32db585-kube-api-access-2j29b\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:47.739279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:47.739286 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042e5e11-0753-4d6d-b606-e0a2a32db585-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:48.325669 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:48.325586 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" Nov 02 15:15:48.325669 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:48.325624 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g" event={"ID":"042e5e11-0753-4d6d-b606-e0a2a32db585","Type":"ContainerDied","Data":"4a03115aa6979e9f2464f8a71b2a9a6bcfb15935598e2f3d16e65a138db9e8c3"} Nov 02 15:15:48.325853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:48.325702 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a03115aa6979e9f2464f8a71b2a9a6bcfb15935598e2f3d16e65a138db9e8c3" Nov 02 15:15:50.153811 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:50.153766 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:15:51.028367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.028333 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f"] Nov 02 15:15:51.028878 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.028846 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" containerID="cri-o://32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396" gracePeriod=30 Nov 02 15:15:51.180987 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.180944 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm"] Nov 02 15:15:51.182251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.182219 5476 cpu_manager.go:414] "containerMap: removing container" podUID="042e5e11-0753-4d6d-b606-e0a2a32db585" containerName="revoke-keys" Nov 02 15:15:51.182251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.182252 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042e5e11-0753-4d6d-b606-e0a2a32db585" containerName="revoke-keys" Nov 02 15:15:51.182471 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.182457 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="042e5e11-0753-4d6d-b606-e0a2a32db585" containerName="revoke-keys" Nov 02 15:15:51.186255 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.186227 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:51.238681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.238650 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:15:51.238847 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.238789 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:15:51.263298 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.263244 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm"] Nov 02 15:15:51.283308 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.283114 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b15ac65d-hpbqm\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:51.283705 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.283605 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b15ac65d-hpbqm\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:51.283908 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.283890 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqmft\" (UniqueName: \"kubernetes.io/projected/d37d2250-b5b6-4807-bd08-cd96c1424e21-kube-api-access-mqmft\") pod \"maas-api-revoke-sub-b15ac65d-hpbqm\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:51.385728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.385685 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b15ac65d-hpbqm\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:51.385942 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.385749 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b15ac65d-hpbqm\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:51.385942 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.385787 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmft\" (UniqueName: \"kubernetes.io/projected/d37d2250-b5b6-4807-bd08-cd96c1424e21-kube-api-access-mqmft\") pod \"maas-api-revoke-sub-b15ac65d-hpbqm\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:51.387444 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.387417 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b15ac65d-hpbqm\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:51.390283 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.390254 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b15ac65d-hpbqm\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:51.443283 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.443248 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqmft\" (UniqueName: \"kubernetes.io/projected/d37d2250-b5b6-4807-bd08-cd96c1424e21-kube-api-access-mqmft\") pod \"maas-api-revoke-sub-b15ac65d-hpbqm\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:51.503903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.503862 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:51.970657 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:51.969725 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm"] Nov 02 15:15:51.975156 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:15:51.975125 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37d2250_b5b6_4807_bd08_cd96c1424e21.slice/crio-92c661d0d431545fa09a364c18e9092ca83ce1c747d6579b78f62a98dc8e5da7 WatchSource:0}: Error finding container 92c661d0d431545fa09a364c18e9092ca83ce1c747d6579b78f62a98dc8e5da7: Status 404 returned error can't find the container with id 92c661d0d431545fa09a364c18e9092ca83ce1c747d6579b78f62a98dc8e5da7 Nov 02 15:15:52.354257 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:52.354223 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" event={"ID":"d37d2250-b5b6-4807-bd08-cd96c1424e21","Type":"ContainerStarted","Data":"92c661d0d431545fa09a364c18e9092ca83ce1c747d6579b78f62a98dc8e5da7"} Nov 02 15:15:53.362819 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:53.362779 5476 generic.go:356] "Generic (PLEG): container finished" podID="d37d2250-b5b6-4807-bd08-cd96c1424e21" containerID="5bb17196b4f661e87af46fa18157be6677581a3697072c5f06147090942ccbf0" exitCode=0 Nov 02 15:15:53.363355 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:53.362855 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" event={"ID":"d37d2250-b5b6-4807-bd08-cd96c1424e21","Type":"ContainerDied","Data":"5bb17196b4f661e87af46fa18157be6677581a3697072c5f06147090942ccbf0"} Nov 02 15:15:54.522331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.522301 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:54.630113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.630022 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-xks-ca\") pod \"d37d2250-b5b6-4807-bd08-cd96c1424e21\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " Nov 02 15:15:54.630470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.630438 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37d2250-b5b6-4807-bd08-cd96c1424e21-kube-api-access-mqmft\" (UniqueName: \"kubernetes.io/projected/d37d2250-b5b6-4807-bd08-cd96c1424e21-kube-api-access-mqmft\") pod \"d37d2250-b5b6-4807-bd08-cd96c1424e21\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " Nov 02 15:15:54.630577 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.630538 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-openshift-ca\") pod \"d37d2250-b5b6-4807-bd08-cd96c1424e21\" (UID: \"d37d2250-b5b6-4807-bd08-cd96c1424e21\") " Nov 02 15:15:54.631312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.631286 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-openshift-ca" pod "d37d2250-b5b6-4807-bd08-cd96c1424e21" (UID: "d37d2250-b5b6-4807-bd08-cd96c1424e21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:54.634052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.634013 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-xks-ca" pod "d37d2250-b5b6-4807-bd08-cd96c1424e21" (UID: "d37d2250-b5b6-4807-bd08-cd96c1424e21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:15:54.634863 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.634816 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37d2250-b5b6-4807-bd08-cd96c1424e21-kube-api-access-mqmft" pod "d37d2250-b5b6-4807-bd08-cd96c1424e21" (UID: "d37d2250-b5b6-4807-bd08-cd96c1424e21"). InnerVolumeSpecName "kube-api-access-mqmft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:54.686798 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.686759 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx"] Nov 02 15:15:54.687347 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.687326 5476 cpu_manager.go:414] "containerMap: removing container" podUID="d37d2250-b5b6-4807-bd08-cd96c1424e21" containerName="revoke-keys" Nov 02 15:15:54.687428 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.687349 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37d2250-b5b6-4807-bd08-cd96c1424e21" containerName="revoke-keys" Nov 02 15:15:54.687464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.687443 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37d2250-b5b6-4807-bd08-cd96c1424e21" containerName="revoke-keys" Nov 02 15:15:54.691116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.691093 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.732087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.732057 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:54.732087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.732082 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqmft\" (UniqueName: \"kubernetes.io/projected/d37d2250-b5b6-4807-bd08-cd96c1424e21-kube-api-access-mqmft\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:54.732087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.732095 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d37d2250-b5b6-4807-bd08-cd96c1424e21-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:54.779926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.779884 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e98976-openshift-default-dockercfg-z9ssr\"" Nov 02 15:15:54.781430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.781399 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx"] Nov 02 15:15:54.833149 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.833117 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.833355 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.833156 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.833355 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.833185 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.833355 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.833307 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8458c4f5-ccc7-4444-b053-61dd0820c344-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.833518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.833416 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.833638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.833616 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.833706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.833659 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlhtx\" (UniqueName: \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-kube-api-access-rlhtx\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.833762 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.833724 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.833813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.833766 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.935337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.935258 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.935337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.935303 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlhtx\" (UniqueName: \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-kube-api-access-rlhtx\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.935557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.935337 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.935557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.935363 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.935557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.935405 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.935557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.935427 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.935557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.935459 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.935557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.935484 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8458c4f5-ccc7-4444-b053-61dd0820c344-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.935557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.935533 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.935902 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.935777 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.936041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.936019 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.936229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.936208 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.936306 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.936262 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.936841 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.936815 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8458c4f5-ccc7-4444-b053-61dd0820c344-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.938683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.938655 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.939324 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.939304 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.940392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.940371 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l"] Nov 02 15:15:54.944522 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.944505 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:54.992884 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.992845 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:54.993414 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.993389 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fce776-openshift-default-dockercfg-5zhzc\"" Nov 02 15:15:54.993590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:54.993567 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlhtx\" (UniqueName: \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-kube-api-access-rlhtx\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:55.005485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.004547 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l"] Nov 02 15:15:55.005485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.004727 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:55.138402 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.138364 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.138605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.138408 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.138605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.138539 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496f7b65-1081-4742-a595-414339df2dee-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.138605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.138579 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsnlx\" (UniqueName: \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-kube-api-access-nsnlx\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.138771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.138618 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.138771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.138723 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.138771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.138760 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.138913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.138785 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.138913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.138868 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496f7b65-1081-4742-a595-414339df2dee-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.239686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.239594 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496f7b65-1081-4742-a595-414339df2dee-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.239853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.239669 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsnlx\" (UniqueName: \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-kube-api-access-nsnlx\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.239853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.239732 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.239853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.239778 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.239853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.239806 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.239853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.239834 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.240140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.239872 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496f7b65-1081-4742-a595-414339df2dee-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.240140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.239920 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.240140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.239946 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.240586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.240299 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.240586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.240386 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.240586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.240512 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.240586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.240553 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.241174 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.241150 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496f7b65-1081-4742-a595-414339df2dee-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.242207 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.242183 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.242328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.242310 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496f7b65-1081-4742-a595-414339df2dee-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.299870 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.299837 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.300054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.299975 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsnlx\" (UniqueName: \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-kube-api-access-nsnlx\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.378480 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.378447 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" event={"ID":"d37d2250-b5b6-4807-bd08-cd96c1424e21","Type":"ContainerDied","Data":"92c661d0d431545fa09a364c18e9092ca83ce1c747d6579b78f62a98dc8e5da7"} Nov 02 15:15:55.378480 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.378477 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm" Nov 02 15:15:55.378652 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.378493 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92c661d0d431545fa09a364c18e9092ca83ce1c747d6579b78f62a98dc8e5da7" Nov 02 15:15:55.451876 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.451836 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx"] Nov 02 15:15:55.459157 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:15:55.459133 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8458c4f5_ccc7_4444_b053_61dd0820c344.slice/crio-c19ac992295057eb6f615fd861dfbbc3db7e522cc907ecbd98fc96c52ca62c3d WatchSource:0}: Error finding container c19ac992295057eb6f615fd861dfbbc3db7e522cc907ecbd98fc96c52ca62c3d: Status 404 returned error can't find the container with id c19ac992295057eb6f615fd861dfbbc3db7e522cc907ecbd98fc96c52ca62c3d Nov 02 15:15:55.461387 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.461358 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:55.461443 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.461419 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:55.461476 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.461447 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:55.562056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.562030 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:55.954824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.954792 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l"] Nov 02 15:15:55.969745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.969182 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:55.969745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.969274 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:55.969745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:55.969318 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:15:56.337767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.337741 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:15:56.354553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.354526 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-certs\") pod \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " Nov 02 15:15:56.354734 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.354558 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-socket\") pod \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " Nov 02 15:15:56.354734 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.354608 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-token\" (UniqueName: \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-token\") pod \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " Nov 02 15:15:56.354734 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.354628 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istiod-ca-cert\") pod \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " Nov 02 15:15:56.354734 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.354674 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-envoy\") pod \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " Nov 02 15:15:56.354734 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.354722 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-data\") pod \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " Nov 02 15:15:56.355013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.354774 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-credential-socket\") pod \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " Nov 02 15:15:56.355013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.354813 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-kube-api-access-g7fh9\" (UniqueName: \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-kube-api-access-g7fh9\") pod \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " Nov 02 15:15:56.355013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.354869 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-certs" pod "493aa9f5-503c-4790-9f0d-d55c7f9757f1" (UID: "493aa9f5-503c-4790-9f0d-d55c7f9757f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:15:56.355013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.354919 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-podinfo\") pod \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\" (UID: \"493aa9f5-503c-4790-9f0d-d55c7f9757f1\") " Nov 02 15:15:56.355221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.354977 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-socket" pod "493aa9f5-503c-4790-9f0d-d55c7f9757f1" (UID: "493aa9f5-503c-4790-9f0d-d55c7f9757f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:15:56.355221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.355129 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-credential-socket" pod "493aa9f5-503c-4790-9f0d-d55c7f9757f1" (UID: "493aa9f5-503c-4790-9f0d-d55c7f9757f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:15:56.355221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.355163 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istiod-ca-cert" pod "493aa9f5-503c-4790-9f0d-d55c7f9757f1" (UID: "493aa9f5-503c-4790-9f0d-d55c7f9757f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:56.355400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.355267 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-data" pod "493aa9f5-503c-4790-9f0d-d55c7f9757f1" (UID: "493aa9f5-503c-4790-9f0d-d55c7f9757f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:15:56.355480 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.355457 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:56.355543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.355485 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:56.355543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.355499 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:56.355543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.355514 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:56.355543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.355527 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:56.357202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.357176 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-token" pod "493aa9f5-503c-4790-9f0d-d55c7f9757f1" (UID: "493aa9f5-503c-4790-9f0d-d55c7f9757f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:56.357202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.357184 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-envoy" pod "493aa9f5-503c-4790-9f0d-d55c7f9757f1" (UID: "493aa9f5-503c-4790-9f0d-d55c7f9757f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:15:56.357429 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.357403 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-kube-api-access-g7fh9" pod "493aa9f5-503c-4790-9f0d-d55c7f9757f1" (UID: "493aa9f5-503c-4790-9f0d-d55c7f9757f1"). InnerVolumeSpecName "kube-api-access-g7fh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:56.357525 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.357510 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-podinfo" pod "493aa9f5-503c-4790-9f0d-d55c7f9757f1" (UID: "493aa9f5-503c-4790-9f0d-d55c7f9757f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:15:56.407860 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.407827 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" event={"ID":"496f7b65-1081-4742-a595-414339df2dee","Type":"ContainerStarted","Data":"0c68d041a88c13b28c52071cc4ef668f1fef398b2caa90bab20652bf7332bdb4"} Nov 02 15:15:56.408079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.407889 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" event={"ID":"496f7b65-1081-4742-a595-414339df2dee","Type":"ContainerStarted","Data":"cf2a2e459ad8d8e8fe56f03bb4e6cecb6d756d6bbf217b0e4e7a6cf06c1b4f4d"} Nov 02 15:15:56.409805 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.409761 5476 generic.go:356] "Generic (PLEG): container finished" podID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerID="32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396" exitCode=0 Nov 02 15:15:56.409930 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.409897 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" event={"ID":"493aa9f5-503c-4790-9f0d-d55c7f9757f1","Type":"ContainerDied","Data":"32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396"} Nov 02 15:15:56.410047 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.409940 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" event={"ID":"493aa9f5-503c-4790-9f0d-d55c7f9757f1","Type":"ContainerDied","Data":"ddee8adff5f8917c2f973abfbcbdf5a24697c77701072ee96ca5c9ad9b4e8797"} Nov 02 15:15:56.410047 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.409972 5476 scope.go:122] "RemoveContainer" containerID="32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396" Nov 02 15:15:56.410184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.410156 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f" Nov 02 15:15:56.418881 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.418750 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" event={"ID":"8458c4f5-ccc7-4444-b053-61dd0820c344","Type":"ContainerStarted","Data":"95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e"} Nov 02 15:15:56.418881 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.418804 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" event={"ID":"8458c4f5-ccc7-4444-b053-61dd0820c344","Type":"ContainerStarted","Data":"c19ac992295057eb6f615fd861dfbbc3db7e522cc907ecbd98fc96c52ca62c3d"} Nov 02 15:15:56.432639 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.432368 5476 scope.go:122] "RemoveContainer" containerID="32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396" Nov 02 15:15:56.433409 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:15:56.433292 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396\": container with ID starting with 32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396 not found: ID does not exist" containerID="32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396" Nov 02 15:15:56.433409 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.433337 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396"} err="failed to get container status \"32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396\": rpc error: code = NotFound desc = could not find container \"32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396\": container with ID starting with 32ec4d4436b0628df89fffc306a178210ba0d2b1b45b12fa5e936b9ab916a396 not found: ID does not exist" Nov 02 15:15:56.457979 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.457390 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:56.457979 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.457424 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:56.457979 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.457464 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7fh9\" (UniqueName: \"kubernetes.io/projected/493aa9f5-503c-4790-9f0d-d55c7f9757f1-kube-api-access-g7fh9\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:56.457979 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.457479 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/493aa9f5-503c-4790-9f0d-d55c7f9757f1-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:56.562844 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.562816 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:15:56.601048 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.598696 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" podStartSLOduration=2.5986765800000002 podStartE2EDuration="2.59867658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:56.596532312 +0000 UTC m=+1058.799410373" watchObservedRunningTime="2026-11-02 15:15:56.59867658 +0000 UTC m=+1058.801554641" Nov 02 15:15:56.708940 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.708902 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f"] Nov 02 15:15:56.721062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.721036 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x665f"] Nov 02 15:15:56.835204 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:56.835090 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" podStartSLOduration=2.8350763260000003 podStartE2EDuration="2.835076326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:56.831532638 +0000 UTC m=+1059.034410692" watchObservedRunningTime="2026-11-02 15:15:56.835076326 +0000 UTC m=+1059.037954385" Nov 02 15:15:57.006163 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:57.006078 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:15:57.018420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:57.018111 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:15:57.018420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:57.018170 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:15:57.563396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:57.563368 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:15:57.563817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:57.563794 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 15:15:57.931144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:57.931115 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" path="/var/lib/kubelet/pods/493aa9f5-503c-4790-9f0d-d55c7f9757f1/volumes" Nov 02 15:15:58.010106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:58.010073 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:15:58.021129 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:58.021093 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:15:58.562691 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:58.562660 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:15:58.562861 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:58.562707 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 15:15:59.026837 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:59.026248 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:15:59.026837 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:59.026371 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:15:59.562962 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:59.562890 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:15:59.563152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:15:59.562976 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 15:16:00.020020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:00.019441 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:16:00.020020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:00.019505 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:16:00.562347 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:00.562316 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:16:00.562737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:00.562367 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 15:16:01.007027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:01.006969 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:16:01.007320 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:01.007288 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:16:01.562380 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:01.562347 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:16:01.562962 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:01.562399 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 15:16:02.005583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.005542 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Nov 02 15:16:02.005774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.005606 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Nov 02 15:16:02.064601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.064564 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj"] Nov 02 15:16:02.065334 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.065307 5476 cpu_manager.go:414] "containerMap: removing container" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" Nov 02 15:16:02.065334 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.065332 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" Nov 02 15:16:02.065497 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.065483 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="493aa9f5-503c-4790-9f0d-d55c7f9757f1" containerName="istio-proxy" Nov 02 15:16:02.070767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.070743 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:02.119695 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.119667 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:02.120041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.120021 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:16:02.121990 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.121966 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj"] Nov 02 15:16:02.210138 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.210107 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5scv\" (UniqueName: \"kubernetes.io/projected/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-kube-api-access-q5scv\") pod \"maas-api-revoke-sub-fba90c25-4lfrj\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:02.210363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.210347 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba90c25-4lfrj\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:02.210479 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.210465 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba90c25-4lfrj\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:02.311514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.311429 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba90c25-4lfrj\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:02.311514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.311474 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba90c25-4lfrj\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:02.311738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.311677 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5scv\" (UniqueName: \"kubernetes.io/projected/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-kube-api-access-q5scv\") pod \"maas-api-revoke-sub-fba90c25-4lfrj\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:02.312328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.312294 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba90c25-4lfrj\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:02.314053 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.314025 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba90c25-4lfrj\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:02.377035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.376639 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5scv\" (UniqueName: \"kubernetes.io/projected/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-kube-api-access-q5scv\") pod \"maas-api-revoke-sub-fba90c25-4lfrj\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:02.408409 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.408373 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:02.562859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.562773 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 15:16:02.562859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.562834 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 15:16:02.801951 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:02.801919 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj"] Nov 02 15:16:02.808236 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:02.808207 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c6dda6_87a3_46e8_8a39_52d8a66d5d77.slice/crio-df2f5082a5bda5c1362f0cc0509052217c49731f68d127bb39246fc041df35a4 WatchSource:0}: Error finding container df2f5082a5bda5c1362f0cc0509052217c49731f68d127bb39246fc041df35a4: Status 404 returned error can't find the container with id df2f5082a5bda5c1362f0cc0509052217c49731f68d127bb39246fc041df35a4 Nov 02 15:16:03.018017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:03.017622 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:16:03.018481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:03.018456 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:16:03.022957 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:03.022940 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:16:03.475773 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:03.475692 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" event={"ID":"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77","Type":"ContainerStarted","Data":"1057ded6d91bd30a6aa00ccd5c8b39e295a856594a12f60c38e03111841288b1"} Nov 02 15:16:03.475773 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:03.475746 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" event={"ID":"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77","Type":"ContainerStarted","Data":"df2f5082a5bda5c1362f0cc0509052217c49731f68d127bb39246fc041df35a4"} Nov 02 15:16:03.566495 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:03.566464 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:16:03.566866 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:03.566729 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:16:03.567381 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:03.567362 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:16:03.587557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:03.587513 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" podStartSLOduration=1.587501974 podStartE2EDuration="1.587501974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:03.587388957 +0000 UTC m=+1065.790267017" watchObservedRunningTime="2026-11-02 15:16:03.587501974 +0000 UTC m=+1065.790380034" Nov 02 15:16:04.484020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:04.483966 5476 generic.go:356] "Generic (PLEG): container finished" podID="b0c6dda6-87a3-46e8-8a39-52d8a66d5d77" containerID="1057ded6d91bd30a6aa00ccd5c8b39e295a856594a12f60c38e03111841288b1" exitCode=0 Nov 02 15:16:04.484197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:04.484035 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" event={"ID":"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77","Type":"ContainerDied","Data":"1057ded6d91bd30a6aa00ccd5c8b39e295a856594a12f60c38e03111841288b1"} Nov 02 15:16:05.623331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:05.623307 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:05.748670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:05.748598 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-xks-ca\") pod \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " Nov 02 15:16:05.748809 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:05.748740 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-openshift-ca\") pod \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " Nov 02 15:16:05.748857 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:05.748814 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-kube-api-access-q5scv\" (UniqueName: \"kubernetes.io/projected/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-kube-api-access-q5scv\") pod \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\" (UID: \"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77\") " Nov 02 15:16:05.749248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:05.749215 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-openshift-ca" pod "b0c6dda6-87a3-46e8-8a39-52d8a66d5d77" (UID: "b0c6dda6-87a3-46e8-8a39-52d8a66d5d77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:05.750589 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:05.750560 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-xks-ca" pod "b0c6dda6-87a3-46e8-8a39-52d8a66d5d77" (UID: "b0c6dda6-87a3-46e8-8a39-52d8a66d5d77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:05.750824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:05.750808 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-kube-api-access-q5scv" pod "b0c6dda6-87a3-46e8-8a39-52d8a66d5d77" (UID: "b0c6dda6-87a3-46e8-8a39-52d8a66d5d77"). InnerVolumeSpecName "kube-api-access-q5scv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:05.849919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:05.849893 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:05.849919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:05.849917 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:05.850092 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:05.849927 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5scv\" (UniqueName: \"kubernetes.io/projected/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77-kube-api-access-q5scv\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:06.497647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.497616 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" Nov 02 15:16:06.497647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.497634 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj" event={"ID":"b0c6dda6-87a3-46e8-8a39-52d8a66d5d77","Type":"ContainerDied","Data":"df2f5082a5bda5c1362f0cc0509052217c49731f68d127bb39246fc041df35a4"} Nov 02 15:16:06.497898 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.497679 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df2f5082a5bda5c1362f0cc0509052217c49731f68d127bb39246fc041df35a4" Nov 02 15:16:06.627668 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.627639 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf"] Nov 02 15:16:06.628151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.628136 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b0c6dda6-87a3-46e8-8a39-52d8a66d5d77" containerName="revoke-keys" Nov 02 15:16:06.628151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.628152 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c6dda6-87a3-46e8-8a39-52d8a66d5d77" containerName="revoke-keys" Nov 02 15:16:06.628268 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.628259 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c6dda6-87a3-46e8-8a39-52d8a66d5d77" containerName="revoke-keys" Nov 02 15:16:06.631510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.631490 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:06.681811 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.681769 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:06.682016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.681984 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:16:06.686100 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.686074 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf"] Nov 02 15:16:06.758853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.758790 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ddvc\" (UniqueName: \"kubernetes.io/projected/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-kube-api-access-4ddvc\") pod \"maas-api-revoke-sub-cb9fc4f7-x5bwf\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:06.758958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.758916 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9fc4f7-x5bwf\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:06.759044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.758971 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9fc4f7-x5bwf\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:06.859586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.859562 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9fc4f7-x5bwf\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:06.859692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.859630 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ddvc\" (UniqueName: \"kubernetes.io/projected/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-kube-api-access-4ddvc\") pod \"maas-api-revoke-sub-cb9fc4f7-x5bwf\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:06.859778 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.859761 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9fc4f7-x5bwf\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:06.860408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.860385 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb9fc4f7-x5bwf\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:06.861854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.861835 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb9fc4f7-x5bwf\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:06.921706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.921674 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ddvc\" (UniqueName: \"kubernetes.io/projected/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-kube-api-access-4ddvc\") pod \"maas-api-revoke-sub-cb9fc4f7-x5bwf\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:06.943923 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:06.943894 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:07.314492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:07.314462 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf"] Nov 02 15:16:07.319650 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:07.319622 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c1dfed_5ec1_4d9c_ab74_8c3700fe4d4c.slice/crio-96f488a08c5bbc3568534f66148b12b061c555d5088e1a4614b39c1ee1968eb9 WatchSource:0}: Error finding container 96f488a08c5bbc3568534f66148b12b061c555d5088e1a4614b39c1ee1968eb9: Status 404 returned error can't find the container with id 96f488a08c5bbc3568534f66148b12b061c555d5088e1a4614b39c1ee1968eb9 Nov 02 15:16:07.505698 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:07.505666 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" event={"ID":"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c","Type":"ContainerStarted","Data":"96f488a08c5bbc3568534f66148b12b061c555d5088e1a4614b39c1ee1968eb9"} Nov 02 15:16:08.513888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:08.513807 5476 generic.go:356] "Generic (PLEG): container finished" podID="55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c" containerID="257c8e55835083bb62785d47557fa3d588b0a4eee937294b80eef44e56bf27fb" exitCode=0 Nov 02 15:16:08.514279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:08.513891 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" event={"ID":"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c","Type":"ContainerDied","Data":"257c8e55835083bb62785d47557fa3d588b0a4eee937294b80eef44e56bf27fb"} Nov 02 15:16:09.657767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:09.657736 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:09.789052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:09.788969 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-kube-api-access-4ddvc\" (UniqueName: \"kubernetes.io/projected/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-kube-api-access-4ddvc\") pod \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " Nov 02 15:16:09.789182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:09.789069 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-openshift-ca\") pod \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " Nov 02 15:16:09.789223 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:09.789184 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-xks-ca\") pod \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\" (UID: \"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c\") " Nov 02 15:16:09.789481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:09.789449 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-openshift-ca" pod "55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c" (UID: "55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:09.789611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:09.789588 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:09.791537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:09.791507 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-xks-ca" pod "55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c" (UID: "55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:09.791630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:09.791532 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-kube-api-access-4ddvc" pod "55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c" (UID: "55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c"). InnerVolumeSpecName "kube-api-access-4ddvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:09.890552 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:09.890523 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:09.890552 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:09.890550 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ddvc\" (UniqueName: \"kubernetes.io/projected/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c-kube-api-access-4ddvc\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:10.526734 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:10.526704 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" Nov 02 15:16:10.526909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:10.526703 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf" event={"ID":"55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c","Type":"ContainerDied","Data":"96f488a08c5bbc3568534f66148b12b061c555d5088e1a4614b39c1ee1968eb9"} Nov 02 15:16:10.526909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:10.526823 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f488a08c5bbc3568534f66148b12b061c555d5088e1a4614b39c1ee1968eb9" Nov 02 15:16:12.673016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:12.670188 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:16:13.757525 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.757481 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk"] Nov 02 15:16:13.758025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.757982 5476 cpu_manager.go:414] "containerMap: removing container" podUID="55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c" containerName="revoke-keys" Nov 02 15:16:13.758025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.758013 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c" containerName="revoke-keys" Nov 02 15:16:13.758173 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.758097 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c" containerName="revoke-keys" Nov 02 15:16:13.761931 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.761902 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:13.825171 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.825143 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:13.825337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.825264 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:16:13.840408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.840380 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ec33e7e-xpmjk\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:13.840570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.840423 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69gx\" (UniqueName: \"kubernetes.io/projected/27b654d9-2735-4565-b4af-222947bbb2d2-kube-api-access-j69gx\") pod \"maas-api-revoke-sub-7ec33e7e-xpmjk\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:13.840645 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.840578 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ec33e7e-xpmjk\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:13.874477 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.874441 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk"] Nov 02 15:16:13.941685 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.941647 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ec33e7e-xpmjk\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:13.941862 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.941734 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69gx\" (UniqueName: \"kubernetes.io/projected/27b654d9-2735-4565-b4af-222947bbb2d2-kube-api-access-j69gx\") pod \"maas-api-revoke-sub-7ec33e7e-xpmjk\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:13.941962 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.941883 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ec33e7e-xpmjk\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:13.945282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.945254 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ec33e7e-xpmjk\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:13.965619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:13.965581 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ec33e7e-xpmjk\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:14.002626 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:14.002596 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69gx\" (UniqueName: \"kubernetes.io/projected/27b654d9-2735-4565-b4af-222947bbb2d2-kube-api-access-j69gx\") pod \"maas-api-revoke-sub-7ec33e7e-xpmjk\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:14.080043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:14.079943 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:14.482175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:14.482023 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk"] Nov 02 15:16:14.597803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:14.597759 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" event={"ID":"27b654d9-2735-4565-b4af-222947bbb2d2","Type":"ContainerStarted","Data":"f6d60bd4856e553584ffbe5a5eb4d422e156b049e38a7c9781e54d23381d27b3"} Nov 02 15:16:15.606659 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:15.606619 5476 generic.go:356] "Generic (PLEG): container finished" podID="27b654d9-2735-4565-b4af-222947bbb2d2" containerID="e18634d7b06434c720941b4d72c0aea3f378924fa4599add6a88ee5728dc1f82" exitCode=0 Nov 02 15:16:15.607189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:15.606697 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" event={"ID":"27b654d9-2735-4565-b4af-222947bbb2d2","Type":"ContainerDied","Data":"e18634d7b06434c720941b4d72c0aea3f378924fa4599add6a88ee5728dc1f82"} Nov 02 15:16:16.754981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:16.754958 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:16.877225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:16.877120 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-xks-ca\") pod \"27b654d9-2735-4565-b4af-222947bbb2d2\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " Nov 02 15:16:16.877225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:16.877203 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b654d9-2735-4565-b4af-222947bbb2d2-kube-api-access-j69gx\" (UniqueName: \"kubernetes.io/projected/27b654d9-2735-4565-b4af-222947bbb2d2-kube-api-access-j69gx\") pod \"27b654d9-2735-4565-b4af-222947bbb2d2\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " Nov 02 15:16:16.877636 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:16.877614 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-openshift-ca\") pod \"27b654d9-2735-4565-b4af-222947bbb2d2\" (UID: \"27b654d9-2735-4565-b4af-222947bbb2d2\") " Nov 02 15:16:16.878062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:16.878038 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-openshift-ca" pod "27b654d9-2735-4565-b4af-222947bbb2d2" (UID: "27b654d9-2735-4565-b4af-222947bbb2d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:16.879536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:16.879505 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b654d9-2735-4565-b4af-222947bbb2d2-kube-api-access-j69gx" pod "27b654d9-2735-4565-b4af-222947bbb2d2" (UID: "27b654d9-2735-4565-b4af-222947bbb2d2"). InnerVolumeSpecName "kube-api-access-j69gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:16.879643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:16.879567 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-xks-ca" pod "27b654d9-2735-4565-b4af-222947bbb2d2" (UID: "27b654d9-2735-4565-b4af-222947bbb2d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:16.979407 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:16.979376 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:16.979407 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:16.979403 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j69gx\" (UniqueName: \"kubernetes.io/projected/27b654d9-2735-4565-b4af-222947bbb2d2-kube-api-access-j69gx\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:16.979407 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:16.979413 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b654d9-2735-4565-b4af-222947bbb2d2-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:17.620143 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:17.620114 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" Nov 02 15:16:17.620329 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:17.620113 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk" event={"ID":"27b654d9-2735-4565-b4af-222947bbb2d2","Type":"ContainerDied","Data":"f6d60bd4856e553584ffbe5a5eb4d422e156b049e38a7c9781e54d23381d27b3"} Nov 02 15:16:17.620329 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:17.620266 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6d60bd4856e553584ffbe5a5eb4d422e156b049e38a7c9781e54d23381d27b3" Nov 02 15:16:19.968207 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:19.968175 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r"] Nov 02 15:16:19.968826 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:19.968806 5476 cpu_manager.go:414] "containerMap: removing container" podUID="27b654d9-2735-4565-b4af-222947bbb2d2" containerName="revoke-keys" Nov 02 15:16:19.968909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:19.968828 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b654d9-2735-4565-b4af-222947bbb2d2" containerName="revoke-keys" Nov 02 15:16:19.969052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:19.969038 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b654d9-2735-4565-b4af-222947bbb2d2" containerName="revoke-keys" Nov 02 15:16:19.973889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:19.973871 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:20.028140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.028112 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r"] Nov 02 15:16:20.070128 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.070105 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:20.070268 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.070249 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:16:20.113728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.113704 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655cm\" (UniqueName: \"kubernetes.io/projected/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-kube-api-access-655cm\") pod \"maas-api-revoke-sub-cdf0b8bb-8s76r\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:20.113876 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.113859 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdf0b8bb-8s76r\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:20.114046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.113933 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdf0b8bb-8s76r\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:20.146301 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:16:20.146268 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:16:20.214615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.214590 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdf0b8bb-8s76r\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:20.214730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.214640 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdf0b8bb-8s76r\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:20.214730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.214689 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655cm\" (UniqueName: \"kubernetes.io/projected/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-kube-api-access-655cm\") pod \"maas-api-revoke-sub-cdf0b8bb-8s76r\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:20.215283 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.215258 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdf0b8bb-8s76r\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:20.216845 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.216823 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdf0b8bb-8s76r\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:20.272734 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.272673 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-655cm\" (UniqueName: \"kubernetes.io/projected/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-kube-api-access-655cm\") pod \"maas-api-revoke-sub-cdf0b8bb-8s76r\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:20.286537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.286518 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:20.659914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:20.659884 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r"] Nov 02 15:16:20.665952 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:20.665926 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c9d245_cd3b_4501_adcc_9ec5b7d1b4a3.slice/crio-7b0b6802a3d22ec1f87775b65cacea1dfb3f142cbdd049a7872f141cc544a6f9 WatchSource:0}: Error finding container 7b0b6802a3d22ec1f87775b65cacea1dfb3f142cbdd049a7872f141cc544a6f9: Status 404 returned error can't find the container with id 7b0b6802a3d22ec1f87775b65cacea1dfb3f142cbdd049a7872f141cc544a6f9 Nov 02 15:16:21.647597 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:21.647561 5476 generic.go:356] "Generic (PLEG): container finished" podID="88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3" containerID="f032dae382975db295aaa022d4a9f28eb81f4051cb396149678b4efe618bb697" exitCode=0 Nov 02 15:16:21.648973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:21.648947 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" event={"ID":"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3","Type":"ContainerDied","Data":"f032dae382975db295aaa022d4a9f28eb81f4051cb396149678b4efe618bb697"} Nov 02 15:16:21.649145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:21.649042 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" event={"ID":"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3","Type":"ContainerStarted","Data":"7b0b6802a3d22ec1f87775b65cacea1dfb3f142cbdd049a7872f141cc544a6f9"} Nov 02 15:16:22.789828 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:22.789803 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:22.845213 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:22.845173 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-openshift-ca\") pod \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " Nov 02 15:16:22.845380 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:22.845337 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-kube-api-access-655cm\" (UniqueName: \"kubernetes.io/projected/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-kube-api-access-655cm\") pod \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " Nov 02 15:16:22.845443 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:22.845374 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-xks-ca\") pod \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\" (UID: \"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3\") " Nov 02 15:16:22.845539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:22.845508 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-openshift-ca" pod "88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3" (UID: "88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:22.845851 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:22.845825 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:22.848395 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:22.848363 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-kube-api-access-655cm" pod "88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3" (UID: "88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3"). InnerVolumeSpecName "kube-api-access-655cm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:22.848472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:22.848460 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-xks-ca" pod "88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3" (UID: "88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:22.946905 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:22.946836 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655cm\" (UniqueName: \"kubernetes.io/projected/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-kube-api-access-655cm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:22.946905 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:22.946862 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:23.661353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:23.661323 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" Nov 02 15:16:23.661353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:23.661349 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r" event={"ID":"88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3","Type":"ContainerDied","Data":"7b0b6802a3d22ec1f87775b65cacea1dfb3f142cbdd049a7872f141cc544a6f9"} Nov 02 15:16:23.661534 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:23.661395 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b0b6802a3d22ec1f87775b65cacea1dfb3f142cbdd049a7872f141cc544a6f9" Nov 02 15:16:26.820266 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.820228 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr"] Nov 02 15:16:26.821049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.821027 5476 cpu_manager.go:414] "containerMap: removing container" podUID="88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3" containerName="revoke-keys" Nov 02 15:16:26.821103 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.821051 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3" containerName="revoke-keys" Nov 02 15:16:26.821242 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.821227 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3" containerName="revoke-keys" Nov 02 15:16:26.825662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.825634 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:26.878735 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.878696 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr"] Nov 02 15:16:26.898162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.898122 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg2g6\" (UniqueName: \"kubernetes.io/projected/9acbf997-dc21-402e-8567-639b5a819aa7-kube-api-access-rg2g6\") pod \"maas-api-revoke-sub-8cc31003-pf6lr\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:26.898290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.898193 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc31003-pf6lr\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:26.898373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.898305 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc31003-pf6lr\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:26.920533 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.920498 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:26.920664 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.920598 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:16:26.999152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.999036 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg2g6\" (UniqueName: \"kubernetes.io/projected/9acbf997-dc21-402e-8567-639b5a819aa7-kube-api-access-rg2g6\") pod \"maas-api-revoke-sub-8cc31003-pf6lr\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:26.999152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.999116 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc31003-pf6lr\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:26.999376 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:26.999159 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc31003-pf6lr\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:27.000254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:27.000224 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc31003-pf6lr\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:27.010795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:27.010767 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc31003-pf6lr\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:27.058144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:27.058122 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg2g6\" (UniqueName: \"kubernetes.io/projected/9acbf997-dc21-402e-8567-639b5a819aa7-kube-api-access-rg2g6\") pod \"maas-api-revoke-sub-8cc31003-pf6lr\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:27.142347 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:27.142320 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:27.523983 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:27.523903 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr"] Nov 02 15:16:27.529435 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:27.529406 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9acbf997_dc21_402e_8567_639b5a819aa7.slice/crio-1db02b0d32eea536ed3673231639e7125b06c039c87ebcd503ab89311c8049fc WatchSource:0}: Error finding container 1db02b0d32eea536ed3673231639e7125b06c039c87ebcd503ab89311c8049fc: Status 404 returned error can't find the container with id 1db02b0d32eea536ed3673231639e7125b06c039c87ebcd503ab89311c8049fc Nov 02 15:16:27.687821 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:27.687781 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" event={"ID":"9acbf997-dc21-402e-8567-639b5a819aa7","Type":"ContainerStarted","Data":"1db02b0d32eea536ed3673231639e7125b06c039c87ebcd503ab89311c8049fc"} Nov 02 15:16:28.705323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:28.699685 5476 generic.go:356] "Generic (PLEG): container finished" podID="9acbf997-dc21-402e-8567-639b5a819aa7" containerID="7bf595a2d82c050336c6d15ddeb773f5af5a3ea8f5e64c3d503c4de1948190f0" exitCode=0 Nov 02 15:16:28.705323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:28.699759 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" event={"ID":"9acbf997-dc21-402e-8567-639b5a819aa7","Type":"ContainerDied","Data":"7bf595a2d82c050336c6d15ddeb773f5af5a3ea8f5e64c3d503c4de1948190f0"} Nov 02 15:16:29.854759 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:29.854735 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:29.929008 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:29.928970 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-openshift-ca\") pod \"9acbf997-dc21-402e-8567-639b5a819aa7\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " Nov 02 15:16:29.929108 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:29.929071 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-xks-ca\") pod \"9acbf997-dc21-402e-8567-639b5a819aa7\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " Nov 02 15:16:29.929157 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:29.929141 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acbf997-dc21-402e-8567-639b5a819aa7-kube-api-access-rg2g6\" (UniqueName: \"kubernetes.io/projected/9acbf997-dc21-402e-8567-639b5a819aa7-kube-api-access-rg2g6\") pod \"9acbf997-dc21-402e-8567-639b5a819aa7\" (UID: \"9acbf997-dc21-402e-8567-639b5a819aa7\") " Nov 02 15:16:29.929309 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:29.929282 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-openshift-ca" pod "9acbf997-dc21-402e-8567-639b5a819aa7" (UID: "9acbf997-dc21-402e-8567-639b5a819aa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:29.929607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:29.929588 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:29.931094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:29.931066 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acbf997-dc21-402e-8567-639b5a819aa7-kube-api-access-rg2g6" pod "9acbf997-dc21-402e-8567-639b5a819aa7" (UID: "9acbf997-dc21-402e-8567-639b5a819aa7"). InnerVolumeSpecName "kube-api-access-rg2g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:29.931186 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:29.931130 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-xks-ca" pod "9acbf997-dc21-402e-8567-639b5a819aa7" (UID: "9acbf997-dc21-402e-8567-639b5a819aa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:30.031136 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.031077 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9acbf997-dc21-402e-8567-639b5a819aa7-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:30.031136 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.031104 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg2g6\" (UniqueName: \"kubernetes.io/projected/9acbf997-dc21-402e-8567-639b5a819aa7-kube-api-access-rg2g6\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:30.035714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.035691 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4"] Nov 02 15:16:30.036323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.036305 5476 cpu_manager.go:414] "containerMap: removing container" podUID="9acbf997-dc21-402e-8567-639b5a819aa7" containerName="revoke-keys" Nov 02 15:16:30.036438 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.036420 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9acbf997-dc21-402e-8567-639b5a819aa7" containerName="revoke-keys" Nov 02 15:16:30.036598 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.036585 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9acbf997-dc21-402e-8567-639b5a819aa7" containerName="revoke-keys" Nov 02 15:16:30.039898 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.039876 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:30.096665 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.096640 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4"] Nov 02 15:16:30.132560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.132525 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0361344-89da-461b-abed-07c1b63d997e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4dcd3cb-7drm4\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:30.132738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.132632 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0361344-89da-461b-abed-07c1b63d997e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4dcd3cb-7drm4\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:30.132738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.132703 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f22r\" (UniqueName: \"kubernetes.io/projected/c0361344-89da-461b-abed-07c1b63d997e-kube-api-access-7f22r\") pod \"maas-api-revoke-sub-d4dcd3cb-7drm4\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:30.233576 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.233532 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f22r\" (UniqueName: \"kubernetes.io/projected/c0361344-89da-461b-abed-07c1b63d997e-kube-api-access-7f22r\") pod \"maas-api-revoke-sub-d4dcd3cb-7drm4\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:30.233712 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.233670 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0361344-89da-461b-abed-07c1b63d997e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4dcd3cb-7drm4\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:30.233800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.233759 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0361344-89da-461b-abed-07c1b63d997e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4dcd3cb-7drm4\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:30.234448 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.234427 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0361344-89da-461b-abed-07c1b63d997e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4dcd3cb-7drm4\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:30.236089 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.236063 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0361344-89da-461b-abed-07c1b63d997e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4dcd3cb-7drm4\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:30.290738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.290692 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f22r\" (UniqueName: \"kubernetes.io/projected/c0361344-89da-461b-abed-07c1b63d997e-kube-api-access-7f22r\") pod \"maas-api-revoke-sub-d4dcd3cb-7drm4\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:30.353885 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.353865 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:30.714319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.714295 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" Nov 02 15:16:30.714465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.714328 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr" event={"ID":"9acbf997-dc21-402e-8567-639b5a819aa7","Type":"ContainerDied","Data":"1db02b0d32eea536ed3673231639e7125b06c039c87ebcd503ab89311c8049fc"} Nov 02 15:16:30.714465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.714377 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db02b0d32eea536ed3673231639e7125b06c039c87ebcd503ab89311c8049fc" Nov 02 15:16:30.732666 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:30.732629 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4"] Nov 02 15:16:30.737549 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:30.737523 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0361344_89da_461b_abed_07c1b63d997e.slice/crio-23130e35dcddfb47efc7dbc2c8a6b462679128311cc267958b5dc5b0c44aac09 WatchSource:0}: Error finding container 23130e35dcddfb47efc7dbc2c8a6b462679128311cc267958b5dc5b0c44aac09: Status 404 returned error can't find the container with id 23130e35dcddfb47efc7dbc2c8a6b462679128311cc267958b5dc5b0c44aac09 Nov 02 15:16:31.721798 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:31.721767 5476 generic.go:356] "Generic (PLEG): container finished" podID="c0361344-89da-461b-abed-07c1b63d997e" containerID="05cb445a491d06367dfd7d3b9b12bd3f7ff48213b9010bc69d5c6ff301d64e53" exitCode=0 Nov 02 15:16:31.722197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:31.721855 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" event={"ID":"c0361344-89da-461b-abed-07c1b63d997e","Type":"ContainerDied","Data":"05cb445a491d06367dfd7d3b9b12bd3f7ff48213b9010bc69d5c6ff301d64e53"} Nov 02 15:16:31.722197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:31.721904 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" event={"ID":"c0361344-89da-461b-abed-07c1b63d997e","Type":"ContainerStarted","Data":"23130e35dcddfb47efc7dbc2c8a6b462679128311cc267958b5dc5b0c44aac09"} Nov 02 15:16:32.873236 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:32.873210 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:32.956737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:32.956708 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0361344-89da-461b-abed-07c1b63d997e-kube-api-access-7f22r\" (UniqueName: \"kubernetes.io/projected/c0361344-89da-461b-abed-07c1b63d997e-kube-api-access-7f22r\") pod \"c0361344-89da-461b-abed-07c1b63d997e\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " Nov 02 15:16:32.956888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:32.956750 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0361344-89da-461b-abed-07c1b63d997e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0361344-89da-461b-abed-07c1b63d997e-maas-api-openshift-ca\") pod \"c0361344-89da-461b-abed-07c1b63d997e\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " Nov 02 15:16:32.956888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:32.956786 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0361344-89da-461b-abed-07c1b63d997e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0361344-89da-461b-abed-07c1b63d997e-maas-api-xks-ca\") pod \"c0361344-89da-461b-abed-07c1b63d997e\" (UID: \"c0361344-89da-461b-abed-07c1b63d997e\") " Nov 02 15:16:32.957170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:32.957150 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0361344-89da-461b-abed-07c1b63d997e-maas-api-openshift-ca" pod "c0361344-89da-461b-abed-07c1b63d997e" (UID: "c0361344-89da-461b-abed-07c1b63d997e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:32.958758 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:32.958733 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0361344-89da-461b-abed-07c1b63d997e-maas-api-xks-ca" pod "c0361344-89da-461b-abed-07c1b63d997e" (UID: "c0361344-89da-461b-abed-07c1b63d997e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:32.958902 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:32.958879 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0361344-89da-461b-abed-07c1b63d997e-kube-api-access-7f22r" pod "c0361344-89da-461b-abed-07c1b63d997e" (UID: "c0361344-89da-461b-abed-07c1b63d997e"). InnerVolumeSpecName "kube-api-access-7f22r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:33.057990 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:33.057908 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f22r\" (UniqueName: \"kubernetes.io/projected/c0361344-89da-461b-abed-07c1b63d997e-kube-api-access-7f22r\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:33.058231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:33.058203 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0361344-89da-461b-abed-07c1b63d997e-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:33.058328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:33.058236 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0361344-89da-461b-abed-07c1b63d997e-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:33.735193 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:33.735161 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" event={"ID":"c0361344-89da-461b-abed-07c1b63d997e","Type":"ContainerDied","Data":"23130e35dcddfb47efc7dbc2c8a6b462679128311cc267958b5dc5b0c44aac09"} Nov 02 15:16:33.735193 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:33.735193 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4" Nov 02 15:16:33.735406 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:33.735207 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23130e35dcddfb47efc7dbc2c8a6b462679128311cc267958b5dc5b0c44aac09" Nov 02 15:16:35.563753 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.563715 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs"] Nov 02 15:16:35.564839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.564814 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c0361344-89da-461b-abed-07c1b63d997e" containerName="revoke-keys" Nov 02 15:16:35.564839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.564840 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0361344-89da-461b-abed-07c1b63d997e" containerName="revoke-keys" Nov 02 15:16:35.565120 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.565104 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0361344-89da-461b-abed-07c1b63d997e" containerName="revoke-keys" Nov 02 15:16:35.570114 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.570092 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:35.622096 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.622070 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs"] Nov 02 15:16:35.664150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.664125 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:35.664292 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.664173 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:16:35.687044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.686987 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rc5m\" (UniqueName: \"kubernetes.io/projected/ee66e224-2fad-4719-bd75-51cd511cb603-kube-api-access-2rc5m\") pod \"maas-api-revoke-sub-4d5e3966-9bzbs\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:35.687176 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.687095 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d5e3966-9bzbs\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:35.687257 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.687236 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d5e3966-9bzbs\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:35.788575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.788537 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d5e3966-9bzbs\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:35.788760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.788613 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rc5m\" (UniqueName: \"kubernetes.io/projected/ee66e224-2fad-4719-bd75-51cd511cb603-kube-api-access-2rc5m\") pod \"maas-api-revoke-sub-4d5e3966-9bzbs\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:35.788760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.788635 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d5e3966-9bzbs\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:35.789407 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.789374 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d5e3966-9bzbs\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:35.790970 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.790948 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d5e3966-9bzbs\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:35.846445 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.846390 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rc5m\" (UniqueName: \"kubernetes.io/projected/ee66e224-2fad-4719-bd75-51cd511cb603-kube-api-access-2rc5m\") pod \"maas-api-revoke-sub-4d5e3966-9bzbs\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:35.884364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:35.884339 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:36.255919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.255888 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs"] Nov 02 15:16:36.259983 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:36.259953 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee66e224_2fad_4719_bd75_51cd511cb603.slice/crio-184dba609b23a9809ff90a06c5aed6eda80838f4cb0b81d8bb068ee4823be2d1 WatchSource:0}: Error finding container 184dba609b23a9809ff90a06c5aed6eda80838f4cb0b81d8bb068ee4823be2d1: Status 404 returned error can't find the container with id 184dba609b23a9809ff90a06c5aed6eda80838f4cb0b81d8bb068ee4823be2d1 Nov 02 15:16:36.387320 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.387292 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b"] Nov 02 15:16:36.392631 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.392606 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.476172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.476140 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b"] Nov 02 15:16:36.497450 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.497419 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.497612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.497478 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wd4q\" (UniqueName: \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-kube-api-access-8wd4q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.497612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.497587 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.497713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.497648 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.497713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.497681 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.497795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.497722 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.497840 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.497816 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.497919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.497906 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.497960 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.497936 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599169 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599142 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599193 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599245 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599309 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wd4q\" (UniqueName: \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-kube-api-access-8wd4q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599374 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599401 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599450 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599534 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599878 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599438 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599878 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599607 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599878 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599661 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.599878 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599734 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.600041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.599934 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.600253 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.600233 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.601815 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.601793 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.602264 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.602235 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.657157 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.657131 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.657475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.657450 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wd4q\" (UniqueName: \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-kube-api-access-8wd4q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-zz92b\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.705589 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.705568 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:36.755744 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.755707 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" event={"ID":"ee66e224-2fad-4719-bd75-51cd511cb603","Type":"ContainerStarted","Data":"847f5a2dc86485350422b96aa291b192f77d8f80d0d9a842aea9627fb531e5ef"} Nov 02 15:16:36.756025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.755782 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" event={"ID":"ee66e224-2fad-4719-bd75-51cd511cb603","Type":"ContainerStarted","Data":"184dba609b23a9809ff90a06c5aed6eda80838f4cb0b81d8bb068ee4823be2d1"} Nov 02 15:16:36.870110 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:36.870065 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" podStartSLOduration=1.870050661 podStartE2EDuration="1.870050661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:36.867346426 +0000 UTC m=+1099.070224487" watchObservedRunningTime="2026-11-02 15:16:36.870050661 +0000 UTC m=+1099.072928721" Nov 02 15:16:37.132715 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.132642 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b"] Nov 02 15:16:37.137259 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:37.137228 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf019ab4f_b2a6_458c_946c_1ec06e443ed7.slice/crio-24131866a2bdf378c0311773d89863a90b6e58852113578f42e1a6eabbb85af0 WatchSource:0}: Error finding container 24131866a2bdf378c0311773d89863a90b6e58852113578f42e1a6eabbb85af0: Status 404 returned error can't find the container with id 24131866a2bdf378c0311773d89863a90b6e58852113578f42e1a6eabbb85af0 Nov 02 15:16:37.139774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.139742 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:37.139874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.139805 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:37.139874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.139834 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:37.636762 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.636725 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2"] Nov 02 15:16:37.640535 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.640519 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:37.724790 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.724713 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2"] Nov 02 15:16:37.764892 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.764859 5476 generic.go:356] "Generic (PLEG): container finished" podID="ee66e224-2fad-4719-bd75-51cd511cb603" containerID="847f5a2dc86485350422b96aa291b192f77d8f80d0d9a842aea9627fb531e5ef" exitCode=0 Nov 02 15:16:37.765102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.764937 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" event={"ID":"ee66e224-2fad-4719-bd75-51cd511cb603","Type":"ContainerDied","Data":"847f5a2dc86485350422b96aa291b192f77d8f80d0d9a842aea9627fb531e5ef"} Nov 02 15:16:37.766837 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.766811 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" event={"ID":"f019ab4f-b2a6-458c-946c-1ec06e443ed7","Type":"ContainerStarted","Data":"47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99"} Nov 02 15:16:37.766964 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.766849 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" event={"ID":"f019ab4f-b2a6-458c-946c-1ec06e443ed7","Type":"ContainerStarted","Data":"24131866a2bdf378c0311773d89863a90b6e58852113578f42e1a6eabbb85af0"} Nov 02 15:16:37.814614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.814574 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmstw\" (UniqueName: \"kubernetes.io/projected/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-kube-api-access-hmstw\") pod \"maas-api-revoke-sub-bbe90abe-pt6c2\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:37.814785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.814750 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbe90abe-pt6c2\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:37.814915 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.814847 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbe90abe-pt6c2\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:37.915823 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.915779 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbe90abe-pt6c2\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:37.916024 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.915859 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbe90abe-pt6c2\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:37.916024 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.915969 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmstw\" (UniqueName: \"kubernetes.io/projected/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-kube-api-access-hmstw\") pod \"maas-api-revoke-sub-bbe90abe-pt6c2\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:37.916689 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.916660 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbe90abe-pt6c2\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:37.918884 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.918852 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbe90abe-pt6c2\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:37.999584 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:37.999461 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" podStartSLOduration=1.999442889 podStartE2EDuration="1.999442889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:37.989755117 +0000 UTC m=+1100.192633179" watchObservedRunningTime="2026-11-02 15:16:37.999442889 +0000 UTC m=+1100.202320949" Nov 02 15:16:38.001427 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.001263 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmstw\" (UniqueName: \"kubernetes.io/projected/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-kube-api-access-hmstw\") pod \"maas-api-revoke-sub-bbe90abe-pt6c2\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:38.253664 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.253165 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:38.611646 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.611609 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj"] Nov 02 15:16:38.616720 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.616697 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:38.669272 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.669171 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2"] Nov 02 15:16:38.702024 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.693657 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj"] Nov 02 15:16:38.706400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.706374 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:38.707713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.707688 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:38.707886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.707856 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:38.725191 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.725164 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4nc9\" (UniqueName: \"kubernetes.io/projected/fe12e16c-99b0-443b-a5c3-10b4919cb675-kube-api-access-n4nc9\") pod \"maas-api-revoke-sub-167a5990-nb5qj\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:38.725336 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.725302 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167a5990-nb5qj\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:38.725407 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.725368 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167a5990-nb5qj\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:38.798134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.798098 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" event={"ID":"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1","Type":"ContainerStarted","Data":"ec98e6c24843884da6d3456b5be10ef8586d65ba2c35e46d08897333df1dfafd"} Nov 02 15:16:38.826376 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.826346 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4nc9\" (UniqueName: \"kubernetes.io/projected/fe12e16c-99b0-443b-a5c3-10b4919cb675-kube-api-access-n4nc9\") pod \"maas-api-revoke-sub-167a5990-nb5qj\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:38.826540 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.826422 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167a5990-nb5qj\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:38.826540 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.826455 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167a5990-nb5qj\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:38.827140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.827111 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-167a5990-nb5qj\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:38.829106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.829086 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-167a5990-nb5qj\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:38.898711 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.898586 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4nc9\" (UniqueName: \"kubernetes.io/projected/fe12e16c-99b0-443b-a5c3-10b4919cb675-kube-api-access-n4nc9\") pod \"maas-api-revoke-sub-167a5990-nb5qj\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:38.930739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:38.930715 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:39.003606 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.003565 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:39.130355 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.130327 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee66e224-2fad-4719-bd75-51cd511cb603-kube-api-access-2rc5m\" (UniqueName: \"kubernetes.io/projected/ee66e224-2fad-4719-bd75-51cd511cb603-kube-api-access-2rc5m\") pod \"ee66e224-2fad-4719-bd75-51cd511cb603\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " Nov 02 15:16:39.130478 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.130388 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-xks-ca\") pod \"ee66e224-2fad-4719-bd75-51cd511cb603\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " Nov 02 15:16:39.130478 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.130472 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-openshift-ca\") pod \"ee66e224-2fad-4719-bd75-51cd511cb603\" (UID: \"ee66e224-2fad-4719-bd75-51cd511cb603\") " Nov 02 15:16:39.130837 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.130808 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-openshift-ca" pod "ee66e224-2fad-4719-bd75-51cd511cb603" (UID: "ee66e224-2fad-4719-bd75-51cd511cb603"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:39.132381 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.132361 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee66e224-2fad-4719-bd75-51cd511cb603-kube-api-access-2rc5m" pod "ee66e224-2fad-4719-bd75-51cd511cb603" (UID: "ee66e224-2fad-4719-bd75-51cd511cb603"). InnerVolumeSpecName "kube-api-access-2rc5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:39.132867 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.132836 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-xks-ca" pod "ee66e224-2fad-4719-bd75-51cd511cb603" (UID: "ee66e224-2fad-4719-bd75-51cd511cb603"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:39.232145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.232072 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rc5m\" (UniqueName: \"kubernetes.io/projected/ee66e224-2fad-4719-bd75-51cd511cb603-kube-api-access-2rc5m\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:39.232145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.232100 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:39.232145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.232109 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee66e224-2fad-4719-bd75-51cd511cb603-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:39.337447 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.337413 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj"] Nov 02 15:16:39.341960 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:39.341934 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe12e16c_99b0_443b_a5c3_10b4919cb675.slice/crio-a5b6c56d30c1ef088044e8a33aa31547e8fed530902066bf81f359f7f19fc595 WatchSource:0}: Error finding container a5b6c56d30c1ef088044e8a33aa31547e8fed530902066bf81f359f7f19fc595: Status 404 returned error can't find the container with id a5b6c56d30c1ef088044e8a33aa31547e8fed530902066bf81f359f7f19fc595 Nov 02 15:16:39.706152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.706125 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:39.706471 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.706178 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:39.806449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.806418 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" Nov 02 15:16:39.806607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.806453 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs" event={"ID":"ee66e224-2fad-4719-bd75-51cd511cb603","Type":"ContainerDied","Data":"184dba609b23a9809ff90a06c5aed6eda80838f4cb0b81d8bb068ee4823be2d1"} Nov 02 15:16:39.806607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.806500 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="184dba609b23a9809ff90a06c5aed6eda80838f4cb0b81d8bb068ee4823be2d1" Nov 02 15:16:39.808300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.808274 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" event={"ID":"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1","Type":"ContainerStarted","Data":"f7a26b070af869ca9bc9fd6b0df5b17eea9d0a95e0d463967b1dc9919c94580d"} Nov 02 15:16:39.810103 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.810075 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" event={"ID":"fe12e16c-99b0-443b-a5c3-10b4919cb675","Type":"ContainerStarted","Data":"cf392f639a069ecddb29ad4468d098aaaca890ba63f273e39c9f9abb6fd96e0b"} Nov 02 15:16:39.810249 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.810118 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" event={"ID":"fe12e16c-99b0-443b-a5c3-10b4919cb675","Type":"ContainerStarted","Data":"a5b6c56d30c1ef088044e8a33aa31547e8fed530902066bf81f359f7f19fc595"} Nov 02 15:16:39.958129 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:39.958081 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" podStartSLOduration=2.9580667160000003 podStartE2EDuration="2.958066716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:39.955730712 +0000 UTC m=+1102.158608773" watchObservedRunningTime="2026-11-02 15:16:39.958066716 +0000 UTC m=+1102.160944783" Nov 02 15:16:40.115349 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:40.115297 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" podStartSLOduration=2.115281177 podStartE2EDuration="2.115281177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:40.114314695 +0000 UTC m=+1102.317192767" watchObservedRunningTime="2026-11-02 15:16:40.115281177 +0000 UTC m=+1102.318159236" Nov 02 15:16:40.340934 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:40.340853 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:16:40.714021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:40.712527 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:40.714021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:40.712601 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:40.835685 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:40.835646 5476 generic.go:356] "Generic (PLEG): container finished" podID="21b1edaf-e52e-42be-9411-ff8fc4a7e8f1" containerID="f7a26b070af869ca9bc9fd6b0df5b17eea9d0a95e0d463967b1dc9919c94580d" exitCode=0 Nov 02 15:16:40.835870 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:40.835755 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" event={"ID":"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1","Type":"ContainerDied","Data":"f7a26b070af869ca9bc9fd6b0df5b17eea9d0a95e0d463967b1dc9919c94580d"} Nov 02 15:16:40.837425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:40.837397 5476 generic.go:356] "Generic (PLEG): container finished" podID="fe12e16c-99b0-443b-a5c3-10b4919cb675" containerID="cf392f639a069ecddb29ad4468d098aaaca890ba63f273e39c9f9abb6fd96e0b" exitCode=0 Nov 02 15:16:40.837552 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:40.837428 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" event={"ID":"fe12e16c-99b0-443b-a5c3-10b4919cb675","Type":"ContainerDied","Data":"cf392f639a069ecddb29ad4468d098aaaca890ba63f273e39c9f9abb6fd96e0b"} Nov 02 15:16:41.706792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:41.706746 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:41.707025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:41.706827 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:42.732078 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.731141 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:42.732078 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.731216 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:42.837020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.836351 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:42.904458 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.902384 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe12e16c-99b0-443b-a5c3-10b4919cb675-kube-api-access-n4nc9\" (UniqueName: \"kubernetes.io/projected/fe12e16c-99b0-443b-a5c3-10b4919cb675-kube-api-access-n4nc9\") pod \"fe12e16c-99b0-443b-a5c3-10b4919cb675\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " Nov 02 15:16:42.904458 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.902499 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-xks-ca\") pod \"fe12e16c-99b0-443b-a5c3-10b4919cb675\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " Nov 02 15:16:42.904458 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.902552 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-openshift-ca\") pod \"fe12e16c-99b0-443b-a5c3-10b4919cb675\" (UID: \"fe12e16c-99b0-443b-a5c3-10b4919cb675\") " Nov 02 15:16:42.904458 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.903272 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-openshift-ca" pod "fe12e16c-99b0-443b-a5c3-10b4919cb675" (UID: "fe12e16c-99b0-443b-a5c3-10b4919cb675"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:42.924418 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.924104 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-xks-ca" pod "fe12e16c-99b0-443b-a5c3-10b4919cb675" (UID: "fe12e16c-99b0-443b-a5c3-10b4919cb675"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:42.930556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.930270 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" event={"ID":"fe12e16c-99b0-443b-a5c3-10b4919cb675","Type":"ContainerDied","Data":"a5b6c56d30c1ef088044e8a33aa31547e8fed530902066bf81f359f7f19fc595"} Nov 02 15:16:42.930556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.930319 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5b6c56d30c1ef088044e8a33aa31547e8fed530902066bf81f359f7f19fc595" Nov 02 15:16:42.930556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.930378 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj" Nov 02 15:16:42.955456 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.942148 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe12e16c-99b0-443b-a5c3-10b4919cb675-kube-api-access-n4nc9" pod "fe12e16c-99b0-443b-a5c3-10b4919cb675" (UID: "fe12e16c-99b0-443b-a5c3-10b4919cb675"). InnerVolumeSpecName "kube-api-access-n4nc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:42.960926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:42.960902 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:43.004416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.004325 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-xks-ca\") pod \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " Nov 02 15:16:43.004678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.004657 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-openshift-ca\") pod \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " Nov 02 15:16:43.004959 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.004940 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-kube-api-access-hmstw\" (UniqueName: \"kubernetes.io/projected/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-kube-api-access-hmstw\") pod \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\" (UID: \"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1\") " Nov 02 15:16:43.005616 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.005592 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:43.012019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.005730 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe12e16c-99b0-443b-a5c3-10b4919cb675-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:43.012160 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.012141 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4nc9\" (UniqueName: \"kubernetes.io/projected/fe12e16c-99b0-443b-a5c3-10b4919cb675-kube-api-access-n4nc9\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:43.030278 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.030248 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-openshift-ca" pod "21b1edaf-e52e-42be-9411-ff8fc4a7e8f1" (UID: "21b1edaf-e52e-42be-9411-ff8fc4a7e8f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:43.030828 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.030805 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-kube-api-access-hmstw" pod "21b1edaf-e52e-42be-9411-ff8fc4a7e8f1" (UID: "21b1edaf-e52e-42be-9411-ff8fc4a7e8f1"). InnerVolumeSpecName "kube-api-access-hmstw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:43.031306 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.031279 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-xks-ca" pod "21b1edaf-e52e-42be-9411-ff8fc4a7e8f1" (UID: "21b1edaf-e52e-42be-9411-ff8fc4a7e8f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:43.135187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.135156 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmstw\" (UniqueName: \"kubernetes.io/projected/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-kube-api-access-hmstw\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:43.135491 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.135473 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:43.135655 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.135639 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:43.706327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.706239 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:43.706327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.706298 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:43.938097 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.938074 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" Nov 02 15:16:43.938411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.938087 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2" event={"ID":"21b1edaf-e52e-42be-9411-ff8fc4a7e8f1","Type":"ContainerDied","Data":"ec98e6c24843884da6d3456b5be10ef8586d65ba2c35e46d08897333df1dfafd"} Nov 02 15:16:43.938411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:43.938138 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec98e6c24843884da6d3456b5be10ef8586d65ba2c35e46d08897333df1dfafd" Nov 02 15:16:44.706602 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:44.706563 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-zz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 15:16:44.706798 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:44.706652 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 15:16:45.856879 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:45.856851 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:45.857504 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:45.857448 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:45.858407 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:45.858389 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:16:50.146453 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:16:50.146416 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:16:54.029926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.029895 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t"] Nov 02 15:16:54.030501 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.030483 5476 cpu_manager.go:414] "containerMap: removing container" podUID="21b1edaf-e52e-42be-9411-ff8fc4a7e8f1" containerName="revoke-keys" Nov 02 15:16:54.030554 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.030504 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b1edaf-e52e-42be-9411-ff8fc4a7e8f1" containerName="revoke-keys" Nov 02 15:16:54.030554 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.030520 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ee66e224-2fad-4719-bd75-51cd511cb603" containerName="revoke-keys" Nov 02 15:16:54.030554 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.030528 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee66e224-2fad-4719-bd75-51cd511cb603" containerName="revoke-keys" Nov 02 15:16:54.030554 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.030552 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fe12e16c-99b0-443b-a5c3-10b4919cb675" containerName="revoke-keys" Nov 02 15:16:54.030670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.030558 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe12e16c-99b0-443b-a5c3-10b4919cb675" containerName="revoke-keys" Nov 02 15:16:54.030728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.030695 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b1edaf-e52e-42be-9411-ff8fc4a7e8f1" containerName="revoke-keys" Nov 02 15:16:54.030728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.030708 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee66e224-2fad-4719-bd75-51cd511cb603" containerName="revoke-keys" Nov 02 15:16:54.030728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.030720 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe12e16c-99b0-443b-a5c3-10b4919cb675" containerName="revoke-keys" Nov 02 15:16:54.035806 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.035787 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:54.084753 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.084726 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:54.130925 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.130897 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:16:54.134155 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.134135 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t"] Nov 02 15:16:54.150154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.150126 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d469335-t964t\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:54.150254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.150241 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d469335-t964t\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:54.150356 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.150342 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfnk\" (UniqueName: \"kubernetes.io/projected/fcf1cd02-304f-4482-9e8b-94d08701aceb-kube-api-access-tzfnk\") pod \"maas-api-revoke-sub-5d469335-t964t\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:54.254667 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.254349 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfnk\" (UniqueName: \"kubernetes.io/projected/fcf1cd02-304f-4482-9e8b-94d08701aceb-kube-api-access-tzfnk\") pod \"maas-api-revoke-sub-5d469335-t964t\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:54.254667 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.254438 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d469335-t964t\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:54.254667 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.254534 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d469335-t964t\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:54.255729 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.255694 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d469335-t964t\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:54.257684 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.257659 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d469335-t964t\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:54.312451 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.312387 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfnk\" (UniqueName: \"kubernetes.io/projected/fcf1cd02-304f-4482-9e8b-94d08701aceb-kube-api-access-tzfnk\") pod \"maas-api-revoke-sub-5d469335-t964t\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:54.349260 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.349236 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:54.721790 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:54.721755 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t"] Nov 02 15:16:54.727726 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:54.727696 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf1cd02_304f_4482_9e8b_94d08701aceb.slice/crio-243cf1ee7159f4934e73d4f220159e3b2229342b4f1f8de320e7a3db37e59716 WatchSource:0}: Error finding container 243cf1ee7159f4934e73d4f220159e3b2229342b4f1f8de320e7a3db37e59716: Status 404 returned error can't find the container with id 243cf1ee7159f4934e73d4f220159e3b2229342b4f1f8de320e7a3db37e59716 Nov 02 15:16:55.007785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.007758 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" event={"ID":"fcf1cd02-304f-4482-9e8b-94d08701aceb","Type":"ContainerStarted","Data":"64bb21e2b950178aefa4b79a8b759025d8da1391942408ea455589f1b0d988c3"} Nov 02 15:16:55.007908 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.007811 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" event={"ID":"fcf1cd02-304f-4482-9e8b-94d08701aceb","Type":"ContainerStarted","Data":"243cf1ee7159f4934e73d4f220159e3b2229342b4f1f8de320e7a3db37e59716"} Nov 02 15:16:55.118365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.118321 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" podStartSLOduration=2.118308871 podStartE2EDuration="2.118308871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:55.118147777 +0000 UTC m=+1117.321025837" watchObservedRunningTime="2026-11-02 15:16:55.118308871 +0000 UTC m=+1117.321186931" Nov 02 15:16:55.141771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.141746 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f"] Nov 02 15:16:55.145808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.145790 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.239073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.231608 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f"] Nov 02 15:16:55.265516 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.265480 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.265689 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.265530 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.265689 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.265560 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.265689 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.265606 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.265689 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.265662 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.265885 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.265716 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkhj4\" (UniqueName: \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-kube-api-access-dkhj4\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.265885 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.265859 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.265980 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.265893 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.265980 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.265912 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.366493 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.366460 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.366662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.366510 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.366662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.366530 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.366662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.366567 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.366662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.366594 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.366662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.366619 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.367032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.366664 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.367032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.366714 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.367032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.366763 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkhj4\" (UniqueName: \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-kube-api-access-dkhj4\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.367032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.366908 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.367032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.366955 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.367338 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.367087 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.367338 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.367179 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.367459 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.367434 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.369153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.369132 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.369461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.369445 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.423869 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.423838 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.423975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.423966 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkhj4\" (UniqueName: \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-kube-api-access-dkhj4\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.458700 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.458680 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:55.816852 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.816776 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw"] Nov 02 15:16:55.820817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.820799 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:55.888673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.888642 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f"] Nov 02 15:16:55.896462 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:55.896412 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c43ec4_5cda_4a28_93a2_b8c12d45c0e9.slice/crio-18d23779cbda51a9141cba90f5ba6754205ce21127b4e4e64a4d2b9e177a8afe WatchSource:0}: Error finding container 18d23779cbda51a9141cba90f5ba6754205ce21127b4e4e64a4d2b9e177a8afe: Status 404 returned error can't find the container with id 18d23779cbda51a9141cba90f5ba6754205ce21127b4e4e64a4d2b9e177a8afe Nov 02 15:16:55.897482 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.897431 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw"] Nov 02 15:16:55.899842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.899803 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:55.899949 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.899889 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:55.899949 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.899941 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:55.974282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.974256 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:55.974397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.974293 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:55.974397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.974377 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:55.974482 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.974416 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:55.974522 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.974495 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:55.974607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.974556 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:55.974666 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.974643 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:55.974727 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.974709 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:55.974780 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:55.974756 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rzxq\" (UniqueName: \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-kube-api-access-8rzxq\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.017551 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.017515 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" event={"ID":"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9","Type":"ContainerStarted","Data":"3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc"} Nov 02 15:16:56.017725 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.017581 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" event={"ID":"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9","Type":"ContainerStarted","Data":"18d23779cbda51a9141cba90f5ba6754205ce21127b4e4e64a4d2b9e177a8afe"} Nov 02 15:16:56.019460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.019435 5476 generic.go:356] "Generic (PLEG): container finished" podID="fcf1cd02-304f-4482-9e8b-94d08701aceb" containerID="64bb21e2b950178aefa4b79a8b759025d8da1391942408ea455589f1b0d988c3" exitCode=0 Nov 02 15:16:56.019587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.019481 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" event={"ID":"fcf1cd02-304f-4482-9e8b-94d08701aceb","Type":"ContainerDied","Data":"64bb21e2b950178aefa4b79a8b759025d8da1391942408ea455589f1b0d988c3"} Nov 02 15:16:56.075932 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.075858 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.075932 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.075890 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.075932 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.075919 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.076157 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.075961 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.076157 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.076041 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.076157 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.076108 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.076157 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.076153 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.076340 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.076214 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.076634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.076607 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.076867 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.076839 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.076941 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.076922 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.077205 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.077184 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.077601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.077582 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.077671 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.076254 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rzxq\" (UniqueName: \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-kube-api-access-8rzxq\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.078953 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.078916 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.079185 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.079168 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.138558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.138396 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" podStartSLOduration=1.13837899 podStartE2EDuration="1.13837899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:56.133043649 +0000 UTC m=+1118.335921711" watchObservedRunningTime="2026-11-02 15:16:56.13837899 +0000 UTC m=+1118.341257051" Nov 02 15:16:56.140986 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.140964 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.143938 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.143917 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rzxq\" (UniqueName: \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-kube-api-access-8rzxq\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.241483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.241453 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6"] Nov 02 15:16:56.247659 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.247634 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.297549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.297523 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-71d1eb-openshift-default-dockercfg-zn6lk\"" Nov 02 15:16:56.305873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.305846 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6"] Nov 02 15:16:56.382494 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.382366 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.382494 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.382454 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.382766 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.382503 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.382766 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.382542 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ca94ef-f26d-483d-b291-11a93cc76fec-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.382766 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.382625 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.382766 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.382655 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.382766 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.382724 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.383065 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.382798 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.383065 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.382822 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjwn\" (UniqueName: \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-kube-api-access-bqjwn\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.433674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.433636 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:56.459788 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.459762 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:16:56.461814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.461790 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:16:56.461913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.461836 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:16:56.498017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.497934 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.498189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.498051 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.498189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.498100 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.498189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.498137 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ca94ef-f26d-483d-b291-11a93cc76fec-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.498373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.498217 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.498373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.498249 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.498373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.498311 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.498528 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.498383 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.498528 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.498410 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjwn\" (UniqueName: \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-kube-api-access-bqjwn\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.499673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.498731 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.499673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.499367 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.499673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.499487 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ca94ef-f26d-483d-b291-11a93cc76fec-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.499905 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.499736 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.500251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.500208 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.504106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.502708 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.504106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.503456 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.558294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.558268 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjwn\" (UniqueName: \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-kube-api-access-bqjwn\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.561786 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.561761 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.562743 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.562717 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:56.899859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.899823 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw"] Nov 02 15:16:56.900873 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:56.900843 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea98dd01_bd7f_4d6e_8988_1c5f70959565.slice/crio-9f886c569db49e0c01cb8726d4510135a61cb588297aa868fe205dd6299dfc69 WatchSource:0}: Error finding container 9f886c569db49e0c01cb8726d4510135a61cb588297aa868fe205dd6299dfc69: Status 404 returned error can't find the container with id 9f886c569db49e0c01cb8726d4510135a61cb588297aa868fe205dd6299dfc69 Nov 02 15:16:56.903418 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.903386 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:56.903523 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.903448 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:56.903523 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.903476 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:57.000075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:56.998013 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6"] Nov 02 15:16:57.001309 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:16:57.001283 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ca94ef_f26d_483d_b291_11a93cc76fec.slice/crio-6e1e73b7a281a2d19cb2dfd37f1c156a385970db9cd05481e8cf0323f2a87c6d WatchSource:0}: Error finding container 6e1e73b7a281a2d19cb2dfd37f1c156a385970db9cd05481e8cf0323f2a87c6d: Status 404 returned error can't find the container with id 6e1e73b7a281a2d19cb2dfd37f1c156a385970db9cd05481e8cf0323f2a87c6d Nov 02 15:16:57.004246 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.004212 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:57.004343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.004284 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:57.004343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.004318 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:57.029742 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.029713 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" event={"ID":"ea98dd01-bd7f-4d6e-8988-1c5f70959565","Type":"ContainerStarted","Data":"0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8"} Nov 02 15:16:57.029863 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.029775 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" event={"ID":"ea98dd01-bd7f-4d6e-8988-1c5f70959565","Type":"ContainerStarted","Data":"9f886c569db49e0c01cb8726d4510135a61cb588297aa868fe205dd6299dfc69"} Nov 02 15:16:57.031467 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.031425 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" event={"ID":"29ca94ef-f26d-483d-b291-11a93cc76fec","Type":"ContainerStarted","Data":"6e1e73b7a281a2d19cb2dfd37f1c156a385970db9cd05481e8cf0323f2a87c6d"} Nov 02 15:16:57.160684 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.160500 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" podStartSLOduration=2.1604792489999998 podStartE2EDuration="2.160479249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:57.148384288 +0000 UTC m=+1119.351262350" watchObservedRunningTime="2026-11-02 15:16:57.160479249 +0000 UTC m=+1119.363357313" Nov 02 15:16:57.242856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.242831 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:57.321810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.321778 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz"] Nov 02 15:16:57.322369 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.322352 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fcf1cd02-304f-4482-9e8b-94d08701aceb" containerName="revoke-keys" Nov 02 15:16:57.322416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.322372 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf1cd02-304f-4482-9e8b-94d08701aceb" containerName="revoke-keys" Nov 02 15:16:57.322467 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.322462 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf1cd02-304f-4482-9e8b-94d08701aceb" containerName="revoke-keys" Nov 02 15:16:57.325737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.325720 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.416507 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.415787 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-624908-openshift-default-dockercfg-zvxm2\"" Nov 02 15:16:57.417200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.417173 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz"] Nov 02 15:16:57.417832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.417804 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-openshift-ca\") pod \"fcf1cd02-304f-4482-9e8b-94d08701aceb\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " Nov 02 15:16:57.417937 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.417889 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf1cd02-304f-4482-9e8b-94d08701aceb-kube-api-access-tzfnk\" (UniqueName: \"kubernetes.io/projected/fcf1cd02-304f-4482-9e8b-94d08701aceb-kube-api-access-tzfnk\") pod \"fcf1cd02-304f-4482-9e8b-94d08701aceb\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " Nov 02 15:16:57.418145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.418124 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-xks-ca\") pod \"fcf1cd02-304f-4482-9e8b-94d08701aceb\" (UID: \"fcf1cd02-304f-4482-9e8b-94d08701aceb\") " Nov 02 15:16:57.418885 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.418860 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-openshift-ca" pod "fcf1cd02-304f-4482-9e8b-94d08701aceb" (UID: "fcf1cd02-304f-4482-9e8b-94d08701aceb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:57.419337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.419306 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.419474 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.419351 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.419474 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.419441 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.419594 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.419483 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.419594 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.419507 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4r8k\" (UniqueName: \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-kube-api-access-k4r8k\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.419755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.419606 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.419755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.419638 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.419755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.419685 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.419755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.419724 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.419965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.419813 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:57.422536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.422468 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-xks-ca" pod "fcf1cd02-304f-4482-9e8b-94d08701aceb" (UID: "fcf1cd02-304f-4482-9e8b-94d08701aceb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:57.422536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.422499 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf1cd02-304f-4482-9e8b-94d08701aceb-kube-api-access-tzfnk" pod "fcf1cd02-304f-4482-9e8b-94d08701aceb" (UID: "fcf1cd02-304f-4482-9e8b-94d08701aceb"). InnerVolumeSpecName "kube-api-access-tzfnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:57.436258 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.435157 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:16:57.436721 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.436429 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:16:57.436721 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.436475 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:16:57.459128 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.459109 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:16:57.459257 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.459231 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:16:57.521386 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.521279 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.521386 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.521329 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.522231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.521661 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.522231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.521712 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.522231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.521783 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.522231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.521809 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.522231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.521857 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.522231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.521906 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.522231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.521936 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.522231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.521951 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4r8k\" (UniqueName: \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-kube-api-access-k4r8k\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.522231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.522078 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcf1cd02-304f-4482-9e8b-94d08701aceb-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:57.522231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.522097 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfnk\" (UniqueName: \"kubernetes.io/projected/fcf1cd02-304f-4482-9e8b-94d08701aceb-kube-api-access-tzfnk\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:57.522833 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.522578 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.522833 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.522643 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.522939 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.522870 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.523191 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.523162 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.524560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.524538 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.524836 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.524816 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.591906 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.591819 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.597019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.595628 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4r8k\" (UniqueName: \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-kube-api-access-k4r8k\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:57.640685 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:57.640649 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:58.040142 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.040054 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" event={"ID":"29ca94ef-f26d-483d-b291-11a93cc76fec","Type":"ContainerStarted","Data":"3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45"} Nov 02 15:16:58.041433 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.041408 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" Nov 02 15:16:58.041553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.041460 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t" event={"ID":"fcf1cd02-304f-4482-9e8b-94d08701aceb","Type":"ContainerDied","Data":"243cf1ee7159f4934e73d4f220159e3b2229342b4f1f8de320e7a3db37e59716"} Nov 02 15:16:58.041553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.041493 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="243cf1ee7159f4934e73d4f220159e3b2229342b4f1f8de320e7a3db37e59716" Nov 02 15:16:58.065249 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.065215 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz"] Nov 02 15:16:58.115057 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.107562 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:58.115057 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.107664 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:58.115057 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.107711 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:58.158773 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.158725 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" podStartSLOduration=2.158709952 podStartE2EDuration="2.158709952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:58.155419732 +0000 UTC m=+1120.358297793" watchObservedRunningTime="2026-11-02 15:16:58.158709952 +0000 UTC m=+1120.361588060" Nov 02 15:16:58.451027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.450184 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:16:58.451027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.450249 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:16:58.459552 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.459519 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:16:58.459821 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.459791 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:16:58.563287 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.563198 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:16:58.564485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.564456 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:16:58.564621 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:58.564508 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:16:59.083018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:59.082225 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" event={"ID":"cc8cd888-ff5b-44a2-b45d-df67e4cf288c","Type":"ContainerStarted","Data":"e3f8d5aeda69cfcd9834788bcc0e358745ce5e0f16960bfa0d1e3df7f735fad2"} Nov 02 15:16:59.083018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:59.082289 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" event={"ID":"cc8cd888-ff5b-44a2-b45d-df67e4cf288c","Type":"ContainerStarted","Data":"969759fc84a6c74c1d14b0696542510caec8fad6707d86040aaf1a1daefa139c"} Nov 02 15:16:59.460017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:59.457583 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:16:59.460017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:59.457669 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:16:59.490042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:59.472152 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:16:59.490042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:59.472228 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:16:59.569020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:59.567927 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:16:59.569020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:59.567982 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:16:59.647020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:59.646044 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:16:59.647020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:59.646554 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:16:59.647020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:16:59.646604 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:00.434523 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:00.434478 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:00.434846 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:00.434813 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:00.465505 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:00.465463 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:00.466144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:00.466109 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:17:00.570043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:00.563418 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:00.570043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:00.563477 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:00.656739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:00.656705 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:00.656987 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:00.656962 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:00.807182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:00.807075 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:01.345017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:01.344088 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:01.446026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:01.445413 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:01.446026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:01.445477 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:01.470296 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:01.470175 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:01.470296 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:01.470235 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:17:01.574510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:01.574388 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:01.574510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:01.574451 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:01.646776 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:01.646631 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:01.646776 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:01.646718 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:02.099068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.088838 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:02.314018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.312030 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:02.356075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.349101 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:02.356075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.349529 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:17:02.504020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.488250 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:02.504020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.488309 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 02 15:17:02.594021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.558273 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:02.594021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.558337 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:02.618068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.610627 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:02.618068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.610685 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:02.618068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.611034 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:02.662067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.644324 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:02.662067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.644402 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:02.662067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.657114 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:02.754024 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:02.716723 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" podStartSLOduration=5.716702947 podStartE2EDuration="5.716702947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:59.246392009 +0000 UTC m=+1121.449270107" watchObservedRunningTime="2026-11-02 15:17:02.716702947 +0000 UTC m=+1124.919581006" Nov 02 15:17:03.619175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:03.619124 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:03.794368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:03.633501 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 15:17:03.889115 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:03.739202 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:03.955035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:03.796323 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:17:03.984532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:03.984486 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:03.984793 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:03.984761 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:04.012028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.011387 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:04.012028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.011439 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:04.012325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.012299 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:04.012418 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.012345 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:04.030771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.030196 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:04.441125 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.441090 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:04.441391 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.441366 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 15:17:04.567580 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.567455 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:04.567580 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.567516 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Nov 02 15:17:04.616470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.616422 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:04.654019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.653448 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:04.654019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.653519 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:04.692075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.691071 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:17:04.692075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.691133 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:04.796541 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.796103 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:17:04.831020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:04.826293 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:17:05.674031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.633412 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:05.674031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.647718 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:17:05.674031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.649316 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:05.674031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.649380 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Nov 02 15:17:05.768020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.751639 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:17:05.768020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.751697 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:05.826627 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.798263 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:05.826627 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.824086 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:05.916353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.908198 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:05.916353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.909385 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:17:06.000251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.964459 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:17:06.000251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:05.964571 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:17:06.089912 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.089882 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:17:06.091834 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.091815 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:17:06.095067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.095050 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:17:06.112177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.112145 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:17:06.123455 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.123403 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:17:06.635126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.635088 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:17:06.635425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.635394 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:06.815600 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.815566 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:17:06.816725 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.816698 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:17:06.817756 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.817733 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:17:06.942227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.942114 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:17:06.943298 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.943276 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:17:06.944158 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:06.944136 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:17:09.457859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.457827 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9"] Nov 02 15:17:09.463080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.463063 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:09.512026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.511984 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:17:09.548456 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.548425 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63294e77-5srt9\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:09.548587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.548561 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqblb\" (UniqueName: \"kubernetes.io/projected/84534f8e-bbbe-4660-b117-9b0025a3d6ed-kube-api-access-nqblb\") pod \"maas-api-revoke-sub-63294e77-5srt9\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:09.548757 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.548739 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63294e77-5srt9\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:09.558031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.557983 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:17:09.561692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.561671 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9"] Nov 02 15:17:09.649355 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.649328 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63294e77-5srt9\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:09.649489 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.649405 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63294e77-5srt9\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:09.649489 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.649464 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqblb\" (UniqueName: \"kubernetes.io/projected/84534f8e-bbbe-4660-b117-9b0025a3d6ed-kube-api-access-nqblb\") pod \"maas-api-revoke-sub-63294e77-5srt9\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:09.649965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.649946 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63294e77-5srt9\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:09.651740 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.651720 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63294e77-5srt9\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:09.708073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.708006 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqblb\" (UniqueName: \"kubernetes.io/projected/84534f8e-bbbe-4660-b117-9b0025a3d6ed-kube-api-access-nqblb\") pod \"maas-api-revoke-sub-63294e77-5srt9\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:09.778137 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:09.778107 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:10.159661 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:10.159631 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9"] Nov 02 15:17:10.161831 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:17:10.161795 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84534f8e_bbbe_4660_b117_9b0025a3d6ed.slice/crio-b2f34376cffd6e3b398f1bb332c073bd6738e85d44c6873c4382278f81fbeb09 WatchSource:0}: Error finding container b2f34376cffd6e3b398f1bb332c073bd6738e85d44c6873c4382278f81fbeb09: Status 404 returned error can't find the container with id b2f34376cffd6e3b398f1bb332c073bd6738e85d44c6873c4382278f81fbeb09 Nov 02 15:17:10.496056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:10.496028 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" event={"ID":"84534f8e-bbbe-4660-b117-9b0025a3d6ed","Type":"ContainerStarted","Data":"2b33ca2bb768cb47b4e489ccde2494f830b84b886f99bcdb6a4346f4dbd44f86"} Nov 02 15:17:10.496370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:10.496082 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" event={"ID":"84534f8e-bbbe-4660-b117-9b0025a3d6ed","Type":"ContainerStarted","Data":"b2f34376cffd6e3b398f1bb332c073bd6738e85d44c6873c4382278f81fbeb09"} Nov 02 15:17:10.610595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:10.610543 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" podStartSLOduration=1.6105289680000001 podStartE2EDuration="1.610528968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:10.60468758 +0000 UTC m=+1132.807565641" watchObservedRunningTime="2026-11-02 15:17:10.610528968 +0000 UTC m=+1132.813407026" Nov 02 15:17:11.503609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:11.503564 5476 generic.go:356] "Generic (PLEG): container finished" podID="84534f8e-bbbe-4660-b117-9b0025a3d6ed" containerID="2b33ca2bb768cb47b4e489ccde2494f830b84b886f99bcdb6a4346f4dbd44f86" exitCode=0 Nov 02 15:17:11.504072 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:11.503651 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" event={"ID":"84534f8e-bbbe-4660-b117-9b0025a3d6ed","Type":"ContainerDied","Data":"2b33ca2bb768cb47b4e489ccde2494f830b84b886f99bcdb6a4346f4dbd44f86"} Nov 02 15:17:12.664104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:12.664078 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:12.785172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:12.785101 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84534f8e-bbbe-4660-b117-9b0025a3d6ed-kube-api-access-nqblb\" (UniqueName: \"kubernetes.io/projected/84534f8e-bbbe-4660-b117-9b0025a3d6ed-kube-api-access-nqblb\") pod \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " Nov 02 15:17:12.785172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:12.785163 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-xks-ca\") pod \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " Nov 02 15:17:12.785371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:12.785204 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-openshift-ca\") pod \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\" (UID: \"84534f8e-bbbe-4660-b117-9b0025a3d6ed\") " Nov 02 15:17:12.785583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:12.785556 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-openshift-ca" pod "84534f8e-bbbe-4660-b117-9b0025a3d6ed" (UID: "84534f8e-bbbe-4660-b117-9b0025a3d6ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:17:12.787284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:12.787258 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-xks-ca" pod "84534f8e-bbbe-4660-b117-9b0025a3d6ed" (UID: "84534f8e-bbbe-4660-b117-9b0025a3d6ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:17:12.787416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:12.787361 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84534f8e-bbbe-4660-b117-9b0025a3d6ed-kube-api-access-nqblb" pod "84534f8e-bbbe-4660-b117-9b0025a3d6ed" (UID: "84534f8e-bbbe-4660-b117-9b0025a3d6ed"). InnerVolumeSpecName "kube-api-access-nqblb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:17:12.886843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:12.886817 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqblb\" (UniqueName: \"kubernetes.io/projected/84534f8e-bbbe-4660-b117-9b0025a3d6ed-kube-api-access-nqblb\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:17:12.886843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:12.886843 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:17:12.886985 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:12.886853 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84534f8e-bbbe-4660-b117-9b0025a3d6ed-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:17:13.517612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:13.517582 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" Nov 02 15:17:13.517777 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:13.517620 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9" event={"ID":"84534f8e-bbbe-4660-b117-9b0025a3d6ed","Type":"ContainerDied","Data":"b2f34376cffd6e3b398f1bb332c073bd6738e85d44c6873c4382278f81fbeb09"} Nov 02 15:17:13.517777 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:13.517666 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2f34376cffd6e3b398f1bb332c073bd6738e85d44c6873c4382278f81fbeb09" Nov 02 15:17:16.143580 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:16.143551 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:17:20.147787 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:17:20.147749 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:17:40.476914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:40.476874 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:17:45.193581 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.193547 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2"] Nov 02 15:17:45.194132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.194116 5476 cpu_manager.go:414] "containerMap: removing container" podUID="84534f8e-bbbe-4660-b117-9b0025a3d6ed" containerName="revoke-keys" Nov 02 15:17:45.194196 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.194132 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84534f8e-bbbe-4660-b117-9b0025a3d6ed" containerName="revoke-keys" Nov 02 15:17:45.194249 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.194225 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="84534f8e-bbbe-4660-b117-9b0025a3d6ed" containerName="revoke-keys" Nov 02 15:17:45.197783 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.197760 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.292714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.292679 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-624908\"/\"distinct-w1-624908-kserve-self-signed-certs\"" Nov 02 15:17:45.296373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.296341 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2"] Nov 02 15:17:45.299055 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.299034 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-home\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.299159 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.299061 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aec73889-564d-462b-9b8b-fa969f995df7-tls-certs\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.299159 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.299080 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djt9z\" (UniqueName: \"kubernetes.io/projected/aec73889-564d-462b-9b8b-fa969f995df7-kube-api-access-djt9z\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.299159 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.299140 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-tmp-dir\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.299275 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.299229 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-dshm\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.299275 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.299265 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-model-cache\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.401079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.401036 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-home\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.401236 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.401097 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aec73889-564d-462b-9b8b-fa969f995df7-tls-certs\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.401236 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.401120 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djt9z\" (UniqueName: \"kubernetes.io/projected/aec73889-564d-462b-9b8b-fa969f995df7-kube-api-access-djt9z\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.401236 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.401148 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-tmp-dir\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.401236 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.401184 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-dshm\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.401236 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.401217 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-model-cache\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.401506 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.401464 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-home\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.401605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.401587 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-tmp-dir\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.401691 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.401673 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-model-cache\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.403455 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.403432 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-dshm\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.403772 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.403755 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aec73889-564d-462b-9b8b-fa969f995df7-tls-certs\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.465759 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.465690 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djt9z\" (UniqueName: \"kubernetes.io/projected/aec73889-564d-462b-9b8b-fa969f995df7-kube-api-access-djt9z\") pod \"distinct-w1-624908-kserve-6c84d5f55f-w2bq2\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.512737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.512705 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:45.943776 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:45.943740 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2"] Nov 02 15:17:45.948358 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:17:45.948332 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaec73889_564d_462b_9b8b_fa969f995df7.slice/crio-f1f0471beea00c48d364d52b40e39828e0b9a6f853a2daf7d38bad40e7323ebe WatchSource:0}: Error finding container f1f0471beea00c48d364d52b40e39828e0b9a6f853a2daf7d38bad40e7323ebe: Status 404 returned error can't find the container with id f1f0471beea00c48d364d52b40e39828e0b9a6f853a2daf7d38bad40e7323ebe Nov 02 15:17:46.742156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:46.742120 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" event={"ID":"aec73889-564d-462b-9b8b-fa969f995df7","Type":"ContainerStarted","Data":"94f16287d6e93e58544e301675d1c07769ba4731f8a7487de92ef7c31e6af05c"} Nov 02 15:17:46.742156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:46.742162 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" event={"ID":"aec73889-564d-462b-9b8b-fa969f995df7","Type":"ContainerStarted","Data":"f1f0471beea00c48d364d52b40e39828e0b9a6f853a2daf7d38bad40e7323ebe"} Nov 02 15:17:46.853394 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:46.853334 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podStartSLOduration=1.8533175499999999 podStartE2EDuration="1.85331755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:46.852483901 +0000 UTC m=+1169.055361961" watchObservedRunningTime="2026-11-02 15:17:46.85331755 +0000 UTC m=+1169.056195611" Nov 02 15:17:47.513399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:47.513344 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:50.146338 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:17:50.146301 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:17:55.513352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:55.513316 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:55.526372 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:55.526344 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:17:55.865177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:17:55.865152 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:18:04.640652 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.640610 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8"] Nov 02 15:18:04.644702 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.644684 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.739776 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.739747 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"distinct-w3-71d1eb-kserve-self-signed-certs\"" Nov 02 15:18:04.743725 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.743698 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8"] Nov 02 15:18:04.764575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.764538 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-home\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.764731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.764669 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-tmp-dir\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.764731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.764716 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-model-cache\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.764825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.764812 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6215819d-1566-43c4-8c45-073d4756a048-tls-certs\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.764865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.764833 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-dshm\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.764924 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.764909 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qt6p\" (UniqueName: \"kubernetes.io/projected/6215819d-1566-43c4-8c45-073d4756a048-kube-api-access-9qt6p\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.866522 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.866488 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qt6p\" (UniqueName: \"kubernetes.io/projected/6215819d-1566-43c4-8c45-073d4756a048-kube-api-access-9qt6p\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.866717 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.866550 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-home\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.866717 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.866607 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-tmp-dir\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.866717 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.866635 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-model-cache\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.866717 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.866701 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6215819d-1566-43c4-8c45-073d4756a048-tls-certs\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.866898 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.866725 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-dshm\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.867039 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.867008 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-home\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.867181 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.867047 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-tmp-dir\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.867181 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.867076 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-model-cache\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.868923 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.868899 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-dshm\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.869239 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.869219 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6215819d-1566-43c4-8c45-073d4756a048-tls-certs\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.924086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.924013 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qt6p\" (UniqueName: \"kubernetes.io/projected/6215819d-1566-43c4-8c45-073d4756a048-kube-api-access-9qt6p\") pod \"distinct-w3-71d1eb-kserve-6cf679899d-mv2s8\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:04.958182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:04.958143 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:05.350328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:05.350244 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8"] Nov 02 15:18:05.354459 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:18:05.354431 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6215819d_1566_43c4_8c45_073d4756a048.slice/crio-661f7e3b75ebde9c025a8bf42d7b02e2793786f1e34b66a31b5f4387692b0f2a WatchSource:0}: Error finding container 661f7e3b75ebde9c025a8bf42d7b02e2793786f1e34b66a31b5f4387692b0f2a: Status 404 returned error can't find the container with id 661f7e3b75ebde9c025a8bf42d7b02e2793786f1e34b66a31b5f4387692b0f2a Nov 02 15:18:05.933454 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:05.933412 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" event={"ID":"6215819d-1566-43c4-8c45-073d4756a048","Type":"ContainerStarted","Data":"56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba"} Nov 02 15:18:05.933454 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:05.933458 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" event={"ID":"6215819d-1566-43c4-8c45-073d4756a048","Type":"ContainerStarted","Data":"661f7e3b75ebde9c025a8bf42d7b02e2793786f1e34b66a31b5f4387692b0f2a"} Nov 02 15:18:05.958317 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:05.958286 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:06.047966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:06.047909 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podStartSLOduration=2.047895862 podStartE2EDuration="2.047895862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:06.043890274 +0000 UTC m=+1188.246768334" watchObservedRunningTime="2026-11-02 15:18:06.047895862 +0000 UTC m=+1188.250773919" Nov 02 15:18:14.363702 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.363661 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7"] Nov 02 15:18:14.367945 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.367925 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.463591 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.463559 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a2e6f\"/\"distinct-2-w4-8a2e6f-kserve-self-signed-certs\"" Nov 02 15:18:14.467357 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.467329 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7"] Nov 02 15:18:14.477302 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.477279 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv2bc\" (UniqueName: \"kubernetes.io/projected/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-kube-api-access-dv2bc\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.477398 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.477310 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-dshm\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.477398 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.477384 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tls-certs\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.477541 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.477523 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-home\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.477578 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.477556 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tmp-dir\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.477674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.477652 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-model-cache\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.578558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.578522 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-home\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.578803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.578566 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tmp-dir\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.578803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.578610 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-model-cache\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.578803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.578680 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv2bc\" (UniqueName: \"kubernetes.io/projected/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-kube-api-access-dv2bc\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.578803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.578708 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-dshm\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.578803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.578782 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tls-certs\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.579112 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.579084 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-home\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.579433 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.579406 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tmp-dir\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.579543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.579440 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-model-cache\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.581361 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.581336 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-dshm\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.581897 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.581876 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tls-certs\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.639965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.639884 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv2bc\" (UniqueName: \"kubernetes.io/projected/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-kube-api-access-dv2bc\") pod \"distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.680640 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.680596 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:14.865217 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.865185 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:18:14.959269 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.959178 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:14.972485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:14.972456 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:15.020495 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:15.020468 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:18:15.063862 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:15.063825 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7"] Nov 02 15:18:15.068436 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:18:15.068410 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecf2b48f_db4b_4720_91e7_d0eaabd1d76c.slice/crio-4797cd3355113b082efd04c2c5b6d2dbaeccb3f4fdf9188abd1736834712b68c WatchSource:0}: Error finding container 4797cd3355113b082efd04c2c5b6d2dbaeccb3f4fdf9188abd1736834712b68c: Status 404 returned error can't find the container with id 4797cd3355113b082efd04c2c5b6d2dbaeccb3f4fdf9188abd1736834712b68c Nov 02 15:18:16.031030 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:16.029883 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" event={"ID":"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c","Type":"ContainerStarted","Data":"018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c"} Nov 02 15:18:16.031030 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:16.029936 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" event={"ID":"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c","Type":"ContainerStarted","Data":"4797cd3355113b082efd04c2c5b6d2dbaeccb3f4fdf9188abd1736834712b68c"} Nov 02 15:18:16.145987 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:16.145927 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podStartSLOduration=2.145913667 podStartE2EDuration="2.145913667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:16.142879203 +0000 UTC m=+1198.345757264" watchObservedRunningTime="2026-11-02 15:18:16.145913667 +0000 UTC m=+1198.348791727" Nov 02 15:18:16.680916 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:16.680874 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:18.084393 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:18.084353 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:18:18.088210 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:18.088184 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:18:20.153209 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:18:20.153159 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:18:24.681368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:24.681326 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:24.694343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:24.694317 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:25.103127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:25.103051 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:18:26.121049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.121013 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8"] Nov 02 15:18:26.125395 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.125370 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.222733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.222698 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8"] Nov 02 15:18:26.225396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.225364 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.225571 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.225520 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9cb32e-03ea-4d07-8763-0b299a061334-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.225644 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.225589 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc89v\" (UniqueName: \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-kube-api-access-nc89v\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.225644 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.225624 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.225754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.225644 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.225754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.225687 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.225754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.225722 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.225914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.225806 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.225914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.225821 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.326510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.326468 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.326714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.326568 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9cb32e-03ea-4d07-8763-0b299a061334-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.326714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.326615 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc89v\" (UniqueName: \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-kube-api-access-nc89v\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.326714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.326653 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.326714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.326680 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.327023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.326739 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.327023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.326775 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.327023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.326880 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.327023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.326905 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.327023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.326953 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.327301 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.327194 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.327374 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.327350 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.327425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.327374 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.327425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.327397 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9cb32e-03ea-4d07-8763-0b299a061334-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.328956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.328937 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.329374 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.329358 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.384730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.384669 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.384917 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.384891 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc89v\" (UniqueName: \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-kube-api-access-nc89v\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.441156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.441131 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:26.873367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.873320 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8"] Nov 02 15:18:26.877291 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:18:26.877270 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9cb32e_03ea_4d07_8763_0b299a061334.slice/crio-8610b865e46796320e7073b297e0ef941ae2c2fb29b5cd8342446e5090893051 WatchSource:0}: Error finding container 8610b865e46796320e7073b297e0ef941ae2c2fb29b5cd8342446e5090893051: Status 404 returned error can't find the container with id 8610b865e46796320e7073b297e0ef941ae2c2fb29b5cd8342446e5090893051 Nov 02 15:18:26.879586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.879556 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:26.879657 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.879619 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:26.879657 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:26.879646 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:27.110052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:27.110012 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" event={"ID":"6c9cb32e-03ea-4d07-8763-0b299a061334","Type":"ContainerStarted","Data":"8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb"} Nov 02 15:18:27.110230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:27.110083 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" event={"ID":"6c9cb32e-03ea-4d07-8763-0b299a061334","Type":"ContainerStarted","Data":"8610b865e46796320e7073b297e0ef941ae2c2fb29b5cd8342446e5090893051"} Nov 02 15:18:27.227839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:27.227749 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" podStartSLOduration=1.227735127 podStartE2EDuration="1.227735127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:27.22666146 +0000 UTC m=+1209.429539521" watchObservedRunningTime="2026-11-02 15:18:27.227735127 +0000 UTC m=+1209.430613187" Nov 02 15:18:27.441914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:27.441876 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:27.443435 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:27.443410 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:18:27.443543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:27.443497 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 15:18:28.441725 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:28.441687 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:18:28.442247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:28.441747 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 15:18:29.442371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:29.442330 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:18:29.442850 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:29.442389 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 15:18:30.441781 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:30.441750 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:18:30.441943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:30.441801 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 15:18:31.442385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:31.442349 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:18:31.442925 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:31.442401 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 15:18:32.441701 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:32.441651 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:18:32.441910 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:32.441718 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 15:18:33.442464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:33.442431 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:18:33.442857 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:33.442489 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 02 15:18:34.445953 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:34.445924 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:34.446362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:34.446219 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:34.447065 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:34.447045 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:18:42.057821 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.057789 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm"] Nov 02 15:18:42.062050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.062031 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.150050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.148875 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.150050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.148928 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.150050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.148977 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.150050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.149110 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlpkt\" (UniqueName: \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-kube-api-access-wlpkt\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.150050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.149161 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.150050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.149214 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.150050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.149257 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.150050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.149344 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.150050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.149480 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.152084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.152055 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm"] Nov 02 15:18:42.251017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.250967 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251025 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251049 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251080 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251143 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlpkt\" (UniqueName: \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-kube-api-access-wlpkt\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251181 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251484 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251230 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251484 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251285 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251484 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251323 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251484 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251416 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251685 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251628 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251814 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.251987 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.251844 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.252103 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.252080 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.253705 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.253679 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.253989 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.253970 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.308612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.308553 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlpkt\" (UniqueName: \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-kube-api-access-wlpkt\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.308612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.308583 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.377455 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.377430 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:42.807197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.807166 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm"] Nov 02 15:18:42.811481 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:18:42.811455 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82cd674_a934_4d35_8ca1_7ba8850d1a0a.slice/crio-7bc3650f5fe2dd2a02491f26e4401f59af918c845b51528f3e94c87e23fbfaf3 WatchSource:0}: Error finding container 7bc3650f5fe2dd2a02491f26e4401f59af918c845b51528f3e94c87e23fbfaf3: Status 404 returned error can't find the container with id 7bc3650f5fe2dd2a02491f26e4401f59af918c845b51528f3e94c87e23fbfaf3 Nov 02 15:18:42.814005 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.813960 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:42.814097 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.814044 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:42.814097 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:42.814077 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:43.230022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:43.229971 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" event={"ID":"c82cd674-a934-4d35-8ca1-7ba8850d1a0a","Type":"ContainerStarted","Data":"b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4"} Nov 02 15:18:43.230418 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:43.230047 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" event={"ID":"c82cd674-a934-4d35-8ca1-7ba8850d1a0a","Type":"ContainerStarted","Data":"7bc3650f5fe2dd2a02491f26e4401f59af918c845b51528f3e94c87e23fbfaf3"} Nov 02 15:18:43.347604 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:43.347546 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" podStartSLOduration=1.347532733 podStartE2EDuration="1.347532733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:43.3454304 +0000 UTC m=+1225.548308473" watchObservedRunningTime="2026-11-02 15:18:43.347532733 +0000 UTC m=+1225.550410792" Nov 02 15:18:43.380296 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:43.380251 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:43.383216 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:43.383179 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:43.383410 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:43.383252 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:44.396136 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:44.396092 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:44.396673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:44.396159 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:44.857275 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:44.857245 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:18:45.381379 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:45.381343 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:45.381770 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:45.381739 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:46.381018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:46.380362 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:46.381018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:46.380443 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:47.390348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:47.390312 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:47.390927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:47.390900 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:48.384224 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:48.384189 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:48.384504 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:48.384475 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:49.378303 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:49.378268 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 02 15:18:49.378692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:49.378331 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 02 15:18:50.149054 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:18:50.148990 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:18:50.381896 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:50.381867 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:50.382310 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:50.382221 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:18:50.383044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:18:50.383028 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:19:10.822390 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:10.822358 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k"] Nov 02 15:19:10.826035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:10.826017 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:10.882556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:10.882525 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k"] Nov 02 15:19:10.923169 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:10.923137 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:19:10.923309 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:10.923174 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:19:10.934345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:10.934321 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56x68\" (UniqueName: \"kubernetes.io/projected/ed438674-a077-43f3-b36f-0c7911ad0b3e-kube-api-access-56x68\") pod \"maas-api-revoke-sub-21c6f771-pkn4k\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:10.934461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:10.934354 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21c6f771-pkn4k\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:10.934461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:10.934372 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21c6f771-pkn4k\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:11.035832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:11.035805 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56x68\" (UniqueName: \"kubernetes.io/projected/ed438674-a077-43f3-b36f-0c7911ad0b3e-kube-api-access-56x68\") pod \"maas-api-revoke-sub-21c6f771-pkn4k\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:11.035963 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:11.035846 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21c6f771-pkn4k\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:11.036046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:11.035972 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21c6f771-pkn4k\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:11.036601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:11.036583 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21c6f771-pkn4k\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:11.038162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:11.038139 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21c6f771-pkn4k\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:11.092938 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:11.092877 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56x68\" (UniqueName: \"kubernetes.io/projected/ed438674-a077-43f3-b36f-0c7911ad0b3e-kube-api-access-56x68\") pod \"maas-api-revoke-sub-21c6f771-pkn4k\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:11.148239 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:11.148220 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:11.522069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:11.522038 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k"] Nov 02 15:19:11.526236 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:19:11.526208 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded438674_a077_43f3_b36f_0c7911ad0b3e.slice/crio-4b86e89137763032d033a4e5f63fb4abc3c75eac5d680325aa2a23d335906060 WatchSource:0}: Error finding container 4b86e89137763032d033a4e5f63fb4abc3c75eac5d680325aa2a23d335906060: Status 404 returned error can't find the container with id 4b86e89137763032d033a4e5f63fb4abc3c75eac5d680325aa2a23d335906060 Nov 02 15:19:12.439128 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:12.439089 5476 generic.go:356] "Generic (PLEG): container finished" podID="ed438674-a077-43f3-b36f-0c7911ad0b3e" containerID="04a5af85c3482050c92d763fd5b435cbedfc7c8dcff58129479d0d0b1efc732a" exitCode=0 Nov 02 15:19:12.439463 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:12.439173 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" event={"ID":"ed438674-a077-43f3-b36f-0c7911ad0b3e","Type":"ContainerDied","Data":"04a5af85c3482050c92d763fd5b435cbedfc7c8dcff58129479d0d0b1efc732a"} Nov 02 15:19:12.439463 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:12.439230 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" event={"ID":"ed438674-a077-43f3-b36f-0c7911ad0b3e","Type":"ContainerStarted","Data":"4b86e89137763032d033a4e5f63fb4abc3c75eac5d680325aa2a23d335906060"} Nov 02 15:19:13.602420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:13.602395 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:13.664960 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:13.664931 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed438674-a077-43f3-b36f-0c7911ad0b3e-kube-api-access-56x68\" (UniqueName: \"kubernetes.io/projected/ed438674-a077-43f3-b36f-0c7911ad0b3e-kube-api-access-56x68\") pod \"ed438674-a077-43f3-b36f-0c7911ad0b3e\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " Nov 02 15:19:13.665107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:13.665044 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-openshift-ca\") pod \"ed438674-a077-43f3-b36f-0c7911ad0b3e\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " Nov 02 15:19:13.665160 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:13.665141 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-xks-ca\") pod \"ed438674-a077-43f3-b36f-0c7911ad0b3e\" (UID: \"ed438674-a077-43f3-b36f-0c7911ad0b3e\") " Nov 02 15:19:13.665363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:13.665337 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-openshift-ca" pod "ed438674-a077-43f3-b36f-0c7911ad0b3e" (UID: "ed438674-a077-43f3-b36f-0c7911ad0b3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:19:13.665604 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:13.665589 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:19:13.667051 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:13.667025 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-xks-ca" pod "ed438674-a077-43f3-b36f-0c7911ad0b3e" (UID: "ed438674-a077-43f3-b36f-0c7911ad0b3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:19:13.667170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:13.667119 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed438674-a077-43f3-b36f-0c7911ad0b3e-kube-api-access-56x68" pod "ed438674-a077-43f3-b36f-0c7911ad0b3e" (UID: "ed438674-a077-43f3-b36f-0c7911ad0b3e"). InnerVolumeSpecName "kube-api-access-56x68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:19:13.766421 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:13.766350 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56x68\" (UniqueName: \"kubernetes.io/projected/ed438674-a077-43f3-b36f-0c7911ad0b3e-kube-api-access-56x68\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:19:13.766421 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:13.766377 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed438674-a077-43f3-b36f-0c7911ad0b3e-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:19:14.463101 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:14.463066 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" Nov 02 15:19:14.463306 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:14.463099 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k" event={"ID":"ed438674-a077-43f3-b36f-0c7911ad0b3e","Type":"ContainerDied","Data":"4b86e89137763032d033a4e5f63fb4abc3c75eac5d680325aa2a23d335906060"} Nov 02 15:19:14.463306 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:14.463172 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b86e89137763032d033a4e5f63fb4abc3c75eac5d680325aa2a23d335906060" Nov 02 15:19:15.649367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.649338 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl"] Nov 02 15:19:15.649860 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.649847 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ed438674-a077-43f3-b36f-0c7911ad0b3e" containerName="revoke-keys" Nov 02 15:19:15.649905 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.649861 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed438674-a077-43f3-b36f-0c7911ad0b3e" containerName="revoke-keys" Nov 02 15:19:15.650010 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.649986 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed438674-a077-43f3-b36f-0c7911ad0b3e" containerName="revoke-keys" Nov 02 15:19:15.653612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.653591 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.750975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.750944 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"kube-root-ca.crt\"" Nov 02 15:19:15.750975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.750944 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"openshift-service-ca.crt\"" Nov 02 15:19:15.751187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.751033 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"default-dockercfg-sw5mb\"" Nov 02 15:19:15.753494 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.753478 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"distinct-2-w5-891723-kserve-self-signed-certs\"" Nov 02 15:19:15.754786 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.754765 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl"] Nov 02 15:19:15.798375 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.798266 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgvcq\" (UniqueName: \"kubernetes.io/projected/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-kube-api-access-rgvcq\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.798697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.798671 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-dshm\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.798865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.798846 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tmp-dir\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.799040 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.799021 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-home\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.799315 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.799290 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-model-cache\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.799427 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.799395 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tls-certs\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.900940 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.900861 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-model-cache\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.900940 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.900928 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tls-certs\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.901156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.900969 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgvcq\" (UniqueName: \"kubernetes.io/projected/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-kube-api-access-rgvcq\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.901156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.901027 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-dshm\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.901156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.901117 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tmp-dir\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.901290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.901201 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-home\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.901379 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.901356 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-model-cache\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.901452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.901430 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tmp-dir\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.901594 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.901572 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-home\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.903224 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.903204 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-dshm\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.903492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.903477 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tls-certs\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.958267 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.958243 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgvcq\" (UniqueName: \"kubernetes.io/projected/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-kube-api-access-rgvcq\") pod \"distinct-2-w5-891723-kserve-557bc4b587-nltfl\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:15.967096 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:15.967064 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:16.353914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:16.353850 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl"] Nov 02 15:19:16.358905 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:19:16.358874 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50036e4d_3d5c_4bb4_8ea8_98193d52cb16.slice/crio-b035191a067d116d92b29e2cfe8c69d736ad6422cf24c95b60a4b5e23b967f5e WatchSource:0}: Error finding container b035191a067d116d92b29e2cfe8c69d736ad6422cf24c95b60a4b5e23b967f5e: Status 404 returned error can't find the container with id b035191a067d116d92b29e2cfe8c69d736ad6422cf24c95b60a4b5e23b967f5e Nov 02 15:19:16.478388 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:16.478358 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" event={"ID":"50036e4d-3d5c-4bb4-8ea8-98193d52cb16","Type":"ContainerStarted","Data":"5f79631293758e9c79dbb8a9782c95632690fe3a63131dfedba7881047c36414"} Nov 02 15:19:16.478513 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:16.478396 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" event={"ID":"50036e4d-3d5c-4bb4-8ea8-98193d52cb16","Type":"ContainerStarted","Data":"b035191a067d116d92b29e2cfe8c69d736ad6422cf24c95b60a4b5e23b967f5e"} Nov 02 15:19:16.597116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:16.597068 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podStartSLOduration=1.597054325 podStartE2EDuration="1.597054325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:16.592932522 +0000 UTC m=+1258.795810581" watchObservedRunningTime="2026-11-02 15:19:16.597054325 +0000 UTC m=+1258.799932385" Nov 02 15:19:16.967897 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:16.967866 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:20.147459 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:19:20.147423 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:19:25.967877 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:25.967845 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:25.980717 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:25.980685 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:26.562221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:26.562192 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:19:27.879123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:27.879050 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:19:39.567293 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.567253 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct"] Nov 02 15:19:39.571328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.571309 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:39.620233 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.620210 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:19:39.666303 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.666276 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:19:39.670126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.670101 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct"] Nov 02 15:19:39.675867 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.675840 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5gq\" (UniqueName: \"kubernetes.io/projected/a7c68892-f972-467d-beab-8db2a842bb8f-kube-api-access-vg5gq\") pod \"maas-api-revoke-sub-608dea81-dtpct\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:39.675973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.675885 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-608dea81-dtpct\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:39.676198 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.676179 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-608dea81-dtpct\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:39.777479 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.777452 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-608dea81-dtpct\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:39.777599 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.777496 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5gq\" (UniqueName: \"kubernetes.io/projected/a7c68892-f972-467d-beab-8db2a842bb8f-kube-api-access-vg5gq\") pod \"maas-api-revoke-sub-608dea81-dtpct\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:39.777599 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.777534 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-608dea81-dtpct\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:39.778233 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.778208 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-608dea81-dtpct\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:39.779760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.779740 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-608dea81-dtpct\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:39.835115 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.835059 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5gq\" (UniqueName: \"kubernetes.io/projected/a7c68892-f972-467d-beab-8db2a842bb8f-kube-api-access-vg5gq\") pod \"maas-api-revoke-sub-608dea81-dtpct\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:39.885380 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:39.885357 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:40.262737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:40.262707 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct"] Nov 02 15:19:40.267779 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:19:40.267752 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c68892_f972_467d_beab_8db2a842bb8f.slice/crio-835665f5bd93ca32f43b5a8d0071181e8310fcc7784f0fe394b68b8dcef959f3 WatchSource:0}: Error finding container 835665f5bd93ca32f43b5a8d0071181e8310fcc7784f0fe394b68b8dcef959f3: Status 404 returned error can't find the container with id 835665f5bd93ca32f43b5a8d0071181e8310fcc7784f0fe394b68b8dcef959f3 Nov 02 15:19:40.654591 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:40.654546 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" event={"ID":"a7c68892-f972-467d-beab-8db2a842bb8f","Type":"ContainerStarted","Data":"830e480c8eb6eaf6209e8bd386b61d984573e70bf36bbfc724a48fa5e9662bd0"} Nov 02 15:19:40.655048 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:40.654611 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" event={"ID":"a7c68892-f972-467d-beab-8db2a842bb8f","Type":"ContainerStarted","Data":"835665f5bd93ca32f43b5a8d0071181e8310fcc7784f0fe394b68b8dcef959f3"} Nov 02 15:19:40.766082 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:40.766030 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" podStartSLOduration=1.766015714 podStartE2EDuration="1.766015714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:40.764790787 +0000 UTC m=+1282.967668846" watchObservedRunningTime="2026-11-02 15:19:40.766015714 +0000 UTC m=+1282.968893774" Nov 02 15:19:41.663485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:41.663450 5476 generic.go:356] "Generic (PLEG): container finished" podID="a7c68892-f972-467d-beab-8db2a842bb8f" containerID="830e480c8eb6eaf6209e8bd386b61d984573e70bf36bbfc724a48fa5e9662bd0" exitCode=0 Nov 02 15:19:41.663882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:41.663535 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" event={"ID":"a7c68892-f972-467d-beab-8db2a842bb8f","Type":"ContainerDied","Data":"830e480c8eb6eaf6209e8bd386b61d984573e70bf36bbfc724a48fa5e9662bd0"} Nov 02 15:19:42.829005 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:42.828973 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:42.912832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:42.912801 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-xks-ca\") pod \"a7c68892-f972-467d-beab-8db2a842bb8f\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " Nov 02 15:19:42.912981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:42.912928 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c68892-f972-467d-beab-8db2a842bb8f-kube-api-access-vg5gq\" (UniqueName: \"kubernetes.io/projected/a7c68892-f972-467d-beab-8db2a842bb8f-kube-api-access-vg5gq\") pod \"a7c68892-f972-467d-beab-8db2a842bb8f\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " Nov 02 15:19:42.913096 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:42.913039 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-openshift-ca\") pod \"a7c68892-f972-467d-beab-8db2a842bb8f\" (UID: \"a7c68892-f972-467d-beab-8db2a842bb8f\") " Nov 02 15:19:42.913332 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:42.913307 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-openshift-ca" pod "a7c68892-f972-467d-beab-8db2a842bb8f" (UID: "a7c68892-f972-467d-beab-8db2a842bb8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:19:42.913449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:42.913430 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:19:42.914865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:42.914839 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-xks-ca" pod "a7c68892-f972-467d-beab-8db2a842bb8f" (UID: "a7c68892-f972-467d-beab-8db2a842bb8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:19:42.914963 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:42.914911 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c68892-f972-467d-beab-8db2a842bb8f-kube-api-access-vg5gq" pod "a7c68892-f972-467d-beab-8db2a842bb8f" (UID: "a7c68892-f972-467d-beab-8db2a842bb8f"). InnerVolumeSpecName "kube-api-access-vg5gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:19:43.015825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:43.015752 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c68892-f972-467d-beab-8db2a842bb8f-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:19:43.015825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:43.015786 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5gq\" (UniqueName: \"kubernetes.io/projected/a7c68892-f972-467d-beab-8db2a842bb8f-kube-api-access-vg5gq\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:19:43.686186 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:43.686158 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" Nov 02 15:19:43.686370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:43.686193 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct" event={"ID":"a7c68892-f972-467d-beab-8db2a842bb8f","Type":"ContainerDied","Data":"835665f5bd93ca32f43b5a8d0071181e8310fcc7784f0fe394b68b8dcef959f3"} Nov 02 15:19:43.686370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:43.686247 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="835665f5bd93ca32f43b5a8d0071181e8310fcc7784f0fe394b68b8dcef959f3" Nov 02 15:19:50.147030 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:19:50.146974 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:19:50.360336 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.360302 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4"] Nov 02 15:19:50.360858 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.360843 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a7c68892-f972-467d-beab-8db2a842bb8f" containerName="revoke-keys" Nov 02 15:19:50.360903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.360860 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c68892-f972-467d-beab-8db2a842bb8f" containerName="revoke-keys" Nov 02 15:19:50.360989 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.360980 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c68892-f972-467d-beab-8db2a842bb8f" containerName="revoke-keys" Nov 02 15:19:50.364660 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.364643 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:50.464553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.464480 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:19:50.464553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.464498 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:19:50.468359 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.468338 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4"] Nov 02 15:19:50.506405 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.506379 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6518b1b3-6rsq4\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:50.506516 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.506492 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvk9\" (UniqueName: \"kubernetes.io/projected/a3487923-57a5-49cd-a252-3dabedca8603-kube-api-access-7zvk9\") pod \"maas-api-revoke-sub-6518b1b3-6rsq4\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:50.506595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.506581 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6518b1b3-6rsq4\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:50.608124 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.608093 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6518b1b3-6rsq4\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:50.608277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.608198 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvk9\" (UniqueName: \"kubernetes.io/projected/a3487923-57a5-49cd-a252-3dabedca8603-kube-api-access-7zvk9\") pod \"maas-api-revoke-sub-6518b1b3-6rsq4\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:50.608342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.608297 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6518b1b3-6rsq4\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:50.608711 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.608692 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6518b1b3-6rsq4\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:50.610357 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.610341 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6518b1b3-6rsq4\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:50.667274 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.667235 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvk9\" (UniqueName: \"kubernetes.io/projected/a3487923-57a5-49cd-a252-3dabedca8603-kube-api-access-7zvk9\") pod \"maas-api-revoke-sub-6518b1b3-6rsq4\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:50.678967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:50.678948 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:51.055961 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:51.055878 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4"] Nov 02 15:19:51.059839 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:19:51.059812 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3487923_57a5_49cd_a252_3dabedca8603.slice/crio-711c6bb91d35b460ba28b53e33c894d1d86785c0346542990ae2b8c4053d77c8 WatchSource:0}: Error finding container 711c6bb91d35b460ba28b53e33c894d1d86785c0346542990ae2b8c4053d77c8: Status 404 returned error can't find the container with id 711c6bb91d35b460ba28b53e33c894d1d86785c0346542990ae2b8c4053d77c8 Nov 02 15:19:51.744831 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:51.744730 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" event={"ID":"a3487923-57a5-49cd-a252-3dabedca8603","Type":"ContainerStarted","Data":"22b7605ba99a347323006c73851a3691f96dfc06f4602e2efa701584524f174c"} Nov 02 15:19:51.744831 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:51.744795 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" event={"ID":"a3487923-57a5-49cd-a252-3dabedca8603","Type":"ContainerStarted","Data":"711c6bb91d35b460ba28b53e33c894d1d86785c0346542990ae2b8c4053d77c8"} Nov 02 15:19:51.857842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:51.857785 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" podStartSLOduration=1.8577692749999999 podStartE2EDuration="1.857769275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:51.854114581 +0000 UTC m=+1294.056992641" watchObservedRunningTime="2026-11-02 15:19:51.857769275 +0000 UTC m=+1294.060647336" Nov 02 15:19:52.753558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:52.753526 5476 generic.go:356] "Generic (PLEG): container finished" podID="a3487923-57a5-49cd-a252-3dabedca8603" containerID="22b7605ba99a347323006c73851a3691f96dfc06f4602e2efa701584524f174c" exitCode=0 Nov 02 15:19:52.753931 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:52.753609 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" event={"ID":"a3487923-57a5-49cd-a252-3dabedca8603","Type":"ContainerDied","Data":"22b7605ba99a347323006c73851a3691f96dfc06f4602e2efa701584524f174c"} Nov 02 15:19:53.923491 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:53.923465 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:54.043683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.043604 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3487923-57a5-49cd-a252-3dabedca8603-kube-api-access-7zvk9\" (UniqueName: \"kubernetes.io/projected/a3487923-57a5-49cd-a252-3dabedca8603-kube-api-access-7zvk9\") pod \"a3487923-57a5-49cd-a252-3dabedca8603\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " Nov 02 15:19:54.043683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.043642 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-openshift-ca\") pod \"a3487923-57a5-49cd-a252-3dabedca8603\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " Nov 02 15:19:54.043909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.043696 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-xks-ca\") pod \"a3487923-57a5-49cd-a252-3dabedca8603\" (UID: \"a3487923-57a5-49cd-a252-3dabedca8603\") " Nov 02 15:19:54.044099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.044068 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-openshift-ca" pod "a3487923-57a5-49cd-a252-3dabedca8603" (UID: "a3487923-57a5-49cd-a252-3dabedca8603"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:19:54.044430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.044405 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:19:54.046112 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.045989 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3487923-57a5-49cd-a252-3dabedca8603-kube-api-access-7zvk9" pod "a3487923-57a5-49cd-a252-3dabedca8603" (UID: "a3487923-57a5-49cd-a252-3dabedca8603"). InnerVolumeSpecName "kube-api-access-7zvk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:19:54.046325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.046301 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-xks-ca" pod "a3487923-57a5-49cd-a252-3dabedca8603" (UID: "a3487923-57a5-49cd-a252-3dabedca8603"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:19:54.145916 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.145891 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvk9\" (UniqueName: \"kubernetes.io/projected/a3487923-57a5-49cd-a252-3dabedca8603-kube-api-access-7zvk9\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:19:54.145916 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.145915 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3487923-57a5-49cd-a252-3dabedca8603-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:19:54.770932 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.770901 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" Nov 02 15:19:54.771188 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.770881 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4" event={"ID":"a3487923-57a5-49cd-a252-3dabedca8603","Type":"ContainerDied","Data":"711c6bb91d35b460ba28b53e33c894d1d86785c0346542990ae2b8c4053d77c8"} Nov 02 15:19:54.771188 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:19:54.771101 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="711c6bb91d35b460ba28b53e33c894d1d86785c0346542990ae2b8c4053d77c8" Nov 02 15:20:01.183289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:01.183252 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:20:07.441023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.440960 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f"] Nov 02 15:20:07.441748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.441726 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a3487923-57a5-49cd-a252-3dabedca8603" containerName="revoke-keys" Nov 02 15:20:07.441850 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.441749 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3487923-57a5-49cd-a252-3dabedca8603" containerName="revoke-keys" Nov 02 15:20:07.441914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.441889 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3487923-57a5-49cd-a252-3dabedca8603" containerName="revoke-keys" Nov 02 15:20:07.445379 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.445359 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.514162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.514122 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fce776\"/\"openshift-service-ca.crt\"" Nov 02 15:20:07.543775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.543717 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fce776\"/\"kube-root-ca.crt\"" Nov 02 15:20:07.543775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.543758 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fce776\"/\"default-dockercfg-fqrg8\"" Nov 02 15:20:07.544163 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.544143 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fce776\"/\"distinct-2-w2-fce776-kserve-self-signed-certs\"" Nov 02 15:20:07.548087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.548065 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f"] Nov 02 15:20:07.553583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.553557 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dgz6\" (UniqueName: \"kubernetes.io/projected/82575b53-acff-42c1-be4c-b863202c8f92-kube-api-access-9dgz6\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.553698 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.553606 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-tmp-dir\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.553756 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.553742 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-dshm\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.553802 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.553790 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-home\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.553847 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.553824 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82575b53-acff-42c1-be4c-b863202c8f92-tls-certs\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.554075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.554052 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-model-cache\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.655834 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.655798 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-dshm\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.656043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.655854 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-home\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.656043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.655888 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82575b53-acff-42c1-be4c-b863202c8f92-tls-certs\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.656043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.656036 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-model-cache\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.656229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.656130 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dgz6\" (UniqueName: \"kubernetes.io/projected/82575b53-acff-42c1-be4c-b863202c8f92-kube-api-access-9dgz6\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.656229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.656171 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-tmp-dir\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.656426 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.656396 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-home\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.656560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.656459 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-model-cache\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.656629 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.656554 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-tmp-dir\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.658410 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.658384 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82575b53-acff-42c1-be4c-b863202c8f92-tls-certs\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.658489 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.658382 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-dshm\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.727240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.727147 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dgz6\" (UniqueName: \"kubernetes.io/projected/82575b53-acff-42c1-be4c-b863202c8f92-kube-api-access-9dgz6\") pod \"distinct-2-w2-fce776-kserve-776d95b978-vgz5f\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:07.757499 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:07.757469 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:08.141610 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.141581 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f"] Nov 02 15:20:08.145078 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:08.145051 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82575b53_acff_42c1_be4c_b863202c8f92.slice/crio-a5951f9a4bd3786dc916ef6f1d9a74091caca0fceba291bed76988a45368c599 WatchSource:0}: Error finding container a5951f9a4bd3786dc916ef6f1d9a74091caca0fceba291bed76988a45368c599: Status 404 returned error can't find the container with id a5951f9a4bd3786dc916ef6f1d9a74091caca0fceba291bed76988a45368c599 Nov 02 15:20:08.727266 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.727231 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf"] Nov 02 15:20:08.731434 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.731417 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:08.788794 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.788768 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf"] Nov 02 15:20:08.829536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.829513 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:20:08.829670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.829544 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:20:08.874694 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.874664 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b175f4c-sjngf\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:08.874881 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.874866 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q8t9\" (UniqueName: \"kubernetes.io/projected/6b628982-17fa-459b-ad3b-881869e3f95f-kube-api-access-6q8t9\") pod \"maas-api-revoke-sub-5b175f4c-sjngf\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:08.874939 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.874918 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b175f4c-sjngf\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:08.907306 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.907279 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" event={"ID":"82575b53-acff-42c1-be4c-b863202c8f92","Type":"ContainerStarted","Data":"709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba"} Nov 02 15:20:08.907438 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.907318 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" event={"ID":"82575b53-acff-42c1-be4c-b863202c8f92","Type":"ContainerStarted","Data":"a5951f9a4bd3786dc916ef6f1d9a74091caca0fceba291bed76988a45368c599"} Nov 02 15:20:08.975726 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.975700 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q8t9\" (UniqueName: \"kubernetes.io/projected/6b628982-17fa-459b-ad3b-881869e3f95f-kube-api-access-6q8t9\") pod \"maas-api-revoke-sub-5b175f4c-sjngf\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:08.975849 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.975761 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b175f4c-sjngf\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:08.975849 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.975800 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b175f4c-sjngf\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:08.976451 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.976428 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b175f4c-sjngf\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:08.978023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:08.977959 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b175f4c-sjngf\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:09.021915 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:09.021876 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podStartSLOduration=2.021863717 podStartE2EDuration="2.021863717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:09.018822925 +0000 UTC m=+1311.221700984" watchObservedRunningTime="2026-11-02 15:20:09.021863717 +0000 UTC m=+1311.224741825" Nov 02 15:20:09.037409 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:09.037378 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q8t9\" (UniqueName: \"kubernetes.io/projected/6b628982-17fa-459b-ad3b-881869e3f95f-kube-api-access-6q8t9\") pod \"maas-api-revoke-sub-5b175f4c-sjngf\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:09.044729 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:09.044709 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:09.433764 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:09.433728 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf"] Nov 02 15:20:09.440075 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:09.440050 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b628982_17fa_459b_ad3b_881869e3f95f.slice/crio-8904dcac158518e04e9804270b62f2fd2d07738ef8de4d33b59222897c4dc85e WatchSource:0}: Error finding container 8904dcac158518e04e9804270b62f2fd2d07738ef8de4d33b59222897c4dc85e: Status 404 returned error can't find the container with id 8904dcac158518e04e9804270b62f2fd2d07738ef8de4d33b59222897c4dc85e Nov 02 15:20:09.758197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:09.758126 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:09.921951 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:09.921912 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" event={"ID":"6b628982-17fa-459b-ad3b-881869e3f95f","Type":"ContainerStarted","Data":"12fa8bd02d9d41ed01458d98a75fc8179ac8f7b0483d09b5d4b8915a8f4f8038"} Nov 02 15:20:09.922104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:09.921965 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" event={"ID":"6b628982-17fa-459b-ad3b-881869e3f95f","Type":"ContainerStarted","Data":"8904dcac158518e04e9804270b62f2fd2d07738ef8de4d33b59222897c4dc85e"} Nov 02 15:20:10.030373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:10.030289 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" podStartSLOduration=2.030275593 podStartE2EDuration="2.030275593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:10.026048027 +0000 UTC m=+1312.228926092" watchObservedRunningTime="2026-11-02 15:20:10.030275593 +0000 UTC m=+1312.233153692" Nov 02 15:20:10.079325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:10.079296 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4"] Nov 02 15:20:10.083101 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:10.083075 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09514c5c-r78f4"] Nov 02 15:20:10.924978 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:10.924943 5476 generic.go:356] "Generic (PLEG): container finished" podID="6b628982-17fa-459b-ad3b-881869e3f95f" containerID="12fa8bd02d9d41ed01458d98a75fc8179ac8f7b0483d09b5d4b8915a8f4f8038" exitCode=0 Nov 02 15:20:10.925396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:10.925042 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" event={"ID":"6b628982-17fa-459b-ad3b-881869e3f95f","Type":"ContainerDied","Data":"12fa8bd02d9d41ed01458d98a75fc8179ac8f7b0483d09b5d4b8915a8f4f8038"} Nov 02 15:20:11.665665 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.665631 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw"] Nov 02 15:20:11.670480 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.670458 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.774372 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.774339 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw"] Nov 02 15:20:11.809845 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.809816 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.809845 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.809849 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.810068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.809973 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4lp\" (UniqueName: \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-kube-api-access-vx4lp\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.810068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.810059 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.810173 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.810114 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.810231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.810203 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.810296 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.810277 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.810400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.810375 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.810516 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.810421 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.911871 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.911843 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912048 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.911902 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912048 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.911946 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912048 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.912037 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912209 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.912184 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912262 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.912249 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912317 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.912294 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912372 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.912319 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912421 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.912386 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912421 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.912392 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4lp\" (UniqueName: \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-kube-api-access-vx4lp\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912658 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.912633 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912735 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.912690 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.912812 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.912877 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.912847 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.914288 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.914273 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.914714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.914693 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.923243 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.923192 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fdc0c2-4270-4cb6-9de4-f90544fb9650" path="/var/lib/kubelet/pods/28fdc0c2-4270-4cb6-9de4-f90544fb9650/volumes" Nov 02 15:20:11.978482 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.978456 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.978925 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.978603 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4lp\" (UniqueName: \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-kube-api-access-vx4lp\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:11.984470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:11.984445 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:12.081054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.075627 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:12.216587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.216504 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b628982-17fa-459b-ad3b-881869e3f95f-kube-api-access-6q8t9\" (UniqueName: \"kubernetes.io/projected/6b628982-17fa-459b-ad3b-881869e3f95f-kube-api-access-6q8t9\") pod \"6b628982-17fa-459b-ad3b-881869e3f95f\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " Nov 02 15:20:12.216753 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.216725 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-openshift-ca\") pod \"6b628982-17fa-459b-ad3b-881869e3f95f\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " Nov 02 15:20:12.216838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.216807 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-xks-ca\") pod \"6b628982-17fa-459b-ad3b-881869e3f95f\" (UID: \"6b628982-17fa-459b-ad3b-881869e3f95f\") " Nov 02 15:20:12.217107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.217079 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-openshift-ca" pod "6b628982-17fa-459b-ad3b-881869e3f95f" (UID: "6b628982-17fa-459b-ad3b-881869e3f95f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:20:12.217391 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.217369 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:12.218769 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.218743 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-xks-ca" pod "6b628982-17fa-459b-ad3b-881869e3f95f" (UID: "6b628982-17fa-459b-ad3b-881869e3f95f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:20:12.219116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.219095 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b628982-17fa-459b-ad3b-881869e3f95f-kube-api-access-6q8t9" pod "6b628982-17fa-459b-ad3b-881869e3f95f" (UID: "6b628982-17fa-459b-ad3b-881869e3f95f"). InnerVolumeSpecName "kube-api-access-6q8t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:20:12.318680 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.318654 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b628982-17fa-459b-ad3b-881869e3f95f-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:12.318680 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.318680 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q8t9\" (UniqueName: \"kubernetes.io/projected/6b628982-17fa-459b-ad3b-881869e3f95f-kube-api-access-6q8t9\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:12.436424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.436140 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw"] Nov 02 15:20:12.440882 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:12.440859 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcde6ebe_5ae4_413f_8cfc_d09877e3e297.slice/crio-6dd8e4ab08d98325f9d145856efab215d3fd8d9682ddb6c1421828637a04282d WatchSource:0}: Error finding container 6dd8e4ab08d98325f9d145856efab215d3fd8d9682ddb6c1421828637a04282d: Status 404 returned error can't find the container with id 6dd8e4ab08d98325f9d145856efab215d3fd8d9682ddb6c1421828637a04282d Nov 02 15:20:12.443357 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.443330 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:12.443423 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.443395 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:12.443469 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.443432 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:12.956816 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.956777 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" event={"ID":"bcde6ebe-5ae4-413f-8cfc-d09877e3e297","Type":"ContainerStarted","Data":"df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9"} Nov 02 15:20:12.956982 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.956868 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" event={"ID":"bcde6ebe-5ae4-413f-8cfc-d09877e3e297","Type":"ContainerStarted","Data":"6dd8e4ab08d98325f9d145856efab215d3fd8d9682ddb6c1421828637a04282d"} Nov 02 15:20:12.958713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.958692 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" Nov 02 15:20:12.958838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.958723 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf" event={"ID":"6b628982-17fa-459b-ad3b-881869e3f95f","Type":"ContainerDied","Data":"8904dcac158518e04e9804270b62f2fd2d07738ef8de4d33b59222897c4dc85e"} Nov 02 15:20:12.958838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.958759 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8904dcac158518e04e9804270b62f2fd2d07738ef8de4d33b59222897c4dc85e" Nov 02 15:20:12.990368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:12.990282 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:13.138609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:13.138560 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" podStartSLOduration=2.138545837 podStartE2EDuration="2.138545837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:13.091329752 +0000 UTC m=+1315.294207808" watchObservedRunningTime="2026-11-02 15:20:13.138545837 +0000 UTC m=+1315.341423896" Nov 02 15:20:13.997023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:13.996965 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:13.997442 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:13.997054 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:14.996153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:14.995650 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:14.996153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:14.995718 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:15.996839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:15.996799 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:15.997365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:15.996858 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:16.993024 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:16.990416 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:16.993024 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:16.990486 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:17.772050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:17.772019 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:17.825494 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:17.825467 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:17.996145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:17.996108 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:17.996336 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:17.996165 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:18.034803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:18.034738 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:18.984976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:18.984944 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 02 15:20:18.985460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:18.985011 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 02 15:20:19.990382 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:19.990350 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:19.990852 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:19.990832 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:19.991611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:19.991593 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:20:20.148649 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:20:20.148613 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:20:24.549944 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.549912 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58"] Nov 02 15:20:24.550538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.550524 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6b628982-17fa-459b-ad3b-881869e3f95f" containerName="revoke-keys" Nov 02 15:20:24.550582 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.550540 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b628982-17fa-459b-ad3b-881869e3f95f" containerName="revoke-keys" Nov 02 15:20:24.550648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.550639 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b628982-17fa-459b-ad3b-881869e3f95f" containerName="revoke-keys" Nov 02 15:20:24.555376 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.555355 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:24.586024 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.585985 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc2dd041-j5l58\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:24.586142 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.586124 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc2dd041-j5l58\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:24.586187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.586153 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gng7\" (UniqueName: \"kubernetes.io/projected/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-kube-api-access-4gng7\") pod \"maas-api-revoke-sub-fc2dd041-j5l58\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:24.610319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.610298 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58"] Nov 02 15:20:24.652056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.652032 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:20:24.652163 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.652066 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:20:24.687284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.687252 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc2dd041-j5l58\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:24.687424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.687291 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gng7\" (UniqueName: \"kubernetes.io/projected/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-kube-api-access-4gng7\") pod \"maas-api-revoke-sub-fc2dd041-j5l58\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:24.687491 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.687458 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc2dd041-j5l58\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:24.688059 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.688031 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc2dd041-j5l58\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:24.689779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.689759 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc2dd041-j5l58\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:24.745725 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.745698 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gng7\" (UniqueName: \"kubernetes.io/projected/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-kube-api-access-4gng7\") pod \"maas-api-revoke-sub-fc2dd041-j5l58\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:24.869073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:24.869043 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:25.245834 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:25.245747 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58"] Nov 02 15:20:25.249354 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:25.249326 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41f318d_c9ac_4d0f_8da2_9fac8931e0ea.slice/crio-0987ed9b300bc24e7154b01c4d595fc3874a95be8e7a9bc00783092eed2dbce5 WatchSource:0}: Error finding container 0987ed9b300bc24e7154b01c4d595fc3874a95be8e7a9bc00783092eed2dbce5: Status 404 returned error can't find the container with id 0987ed9b300bc24e7154b01c4d595fc3874a95be8e7a9bc00783092eed2dbce5 Nov 02 15:20:26.082065 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:26.082026 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" event={"ID":"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea","Type":"ContainerStarted","Data":"5d5f72ab477cb931cd34801c8194411c56c10b52a4730b0807250ca8e5705342"} Nov 02 15:20:26.082528 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:26.082114 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" event={"ID":"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea","Type":"ContainerStarted","Data":"0987ed9b300bc24e7154b01c4d595fc3874a95be8e7a9bc00783092eed2dbce5"} Nov 02 15:20:26.199184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:26.199138 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" podStartSLOduration=2.199123629 podStartE2EDuration="2.199123629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:26.196866361 +0000 UTC m=+1328.399744421" watchObservedRunningTime="2026-11-02 15:20:26.199123629 +0000 UTC m=+1328.402001688" Nov 02 15:20:27.090678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.090638 5476 generic.go:356] "Generic (PLEG): container finished" podID="b41f318d-c9ac-4d0f-8da2-9fac8931e0ea" containerID="5d5f72ab477cb931cd34801c8194411c56c10b52a4730b0807250ca8e5705342" exitCode=0 Nov 02 15:20:27.091091 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.090700 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" event={"ID":"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea","Type":"ContainerDied","Data":"5d5f72ab477cb931cd34801c8194411c56c10b52a4730b0807250ca8e5705342"} Nov 02 15:20:27.469187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.469154 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq"] Nov 02 15:20:27.473387 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.473366 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.521572 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.521540 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62a3481-0bbf-4d16-83f0-524d84b5964c-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.521748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.521718 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.521872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.521771 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.521872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.521824 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsh4k\" (UniqueName: \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-kube-api-access-qsh4k\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.522064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.521877 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.522064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.521981 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.522064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.522032 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.522221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.522145 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.522221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.522174 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.593570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.593519 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq"] Nov 02 15:20:27.623803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.623775 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.623978 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.623828 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.623978 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.623942 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.624135 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.624064 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.624135 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.624088 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.624135 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.624127 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62a3481-0bbf-4d16-83f0-524d84b5964c-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.624312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.624219 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.624312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.624254 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.624424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.624329 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsh4k\" (UniqueName: \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-kube-api-access-qsh4k\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.624424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.624334 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.624424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.624412 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.624592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.624559 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.624645 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.624627 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.624929 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.624906 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62a3481-0bbf-4d16-83f0-524d84b5964c-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.626121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.626103 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.626644 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.626622 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.697859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.697828 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.698088 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.698066 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsh4k\" (UniqueName: \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-kube-api-access-qsh4k\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:27.787384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:27.787299 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:28.221557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.221532 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:28.235911 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.235883 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq"] Nov 02 15:20:28.239500 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:28.239471 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62a3481_0bbf_4d16_83f0_524d84b5964c.slice/crio-db26a69f6cc5493bf6c5252a2eda24038b0c3993f961f327a6acb9fd9862155e WatchSource:0}: Error finding container db26a69f6cc5493bf6c5252a2eda24038b0c3993f961f327a6acb9fd9862155e: Status 404 returned error can't find the container with id db26a69f6cc5493bf6c5252a2eda24038b0c3993f961f327a6acb9fd9862155e Nov 02 15:20:28.241861 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.241822 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.241949 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.241896 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.241949 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.241931 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.333321 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.333294 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-openshift-ca\") pod \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " Nov 02 15:20:28.333560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.333538 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-xks-ca\") pod \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " Nov 02 15:20:28.333623 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.333612 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-kube-api-access-4gng7\" (UniqueName: \"kubernetes.io/projected/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-kube-api-access-4gng7\") pod \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\" (UID: \"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea\") " Nov 02 15:20:28.333677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.333618 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-openshift-ca" pod "b41f318d-c9ac-4d0f-8da2-9fac8931e0ea" (UID: "b41f318d-c9ac-4d0f-8da2-9fac8931e0ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:20:28.334095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.334071 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:28.335982 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.335959 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-xks-ca" pod "b41f318d-c9ac-4d0f-8da2-9fac8931e0ea" (UID: "b41f318d-c9ac-4d0f-8da2-9fac8931e0ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:20:28.336416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.336389 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-kube-api-access-4gng7" pod "b41f318d-c9ac-4d0f-8da2-9fac8931e0ea" (UID: "b41f318d-c9ac-4d0f-8da2-9fac8931e0ea"). InnerVolumeSpecName "kube-api-access-4gng7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:20:28.434865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.434836 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:28.434865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:28.434863 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gng7\" (UniqueName: \"kubernetes.io/projected/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea-kube-api-access-4gng7\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:29.108128 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:29.108085 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" event={"ID":"d62a3481-0bbf-4d16-83f0-524d84b5964c","Type":"ContainerStarted","Data":"2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e"} Nov 02 15:20:29.108316 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:29.108153 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" event={"ID":"d62a3481-0bbf-4d16-83f0-524d84b5964c","Type":"ContainerStarted","Data":"db26a69f6cc5493bf6c5252a2eda24038b0c3993f961f327a6acb9fd9862155e"} Nov 02 15:20:29.109555 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:29.109536 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" Nov 02 15:20:29.109638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:29.109551 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58" event={"ID":"b41f318d-c9ac-4d0f-8da2-9fac8931e0ea","Type":"ContainerDied","Data":"0987ed9b300bc24e7154b01c4d595fc3874a95be8e7a9bc00783092eed2dbce5"} Nov 02 15:20:29.109638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:29.109588 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0987ed9b300bc24e7154b01c4d595fc3874a95be8e7a9bc00783092eed2dbce5" Nov 02 15:20:29.317395 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:29.317344 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" podStartSLOduration=2.317329064 podStartE2EDuration="2.317329064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:29.313008321 +0000 UTC m=+1331.515886374" watchObservedRunningTime="2026-11-02 15:20:29.317329064 +0000 UTC m=+1331.520207122" Nov 02 15:20:29.792017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:29.788115 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:29.794731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:29.792628 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:20:29.794731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:29.792696 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 15:20:30.792019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:30.790462 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:20:30.792019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:30.790534 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 15:20:31.799020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:31.790874 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:20:31.799020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:31.790935 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 15:20:32.189018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:32.180085 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp"] Nov 02 15:20:32.307018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:32.304058 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40383841-8pgxp"] Nov 02 15:20:32.932020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:32.914136 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:20:32.932020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:32.914199 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 15:20:32.932020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:32.915148 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 15:20:32.932020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:32.915476 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:20:32.932020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:32.921696 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.932020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:32.921985 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:20:33.035025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.032527 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.035025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.032655 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 15:20:33.191090 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.175386 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.191090 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.176850 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:20:33.191090 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.182475 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:20:33.320668 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.320625 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:20:33.581077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.576442 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.698153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.698110 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.791018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.790817 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:20:33.791018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.790882 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 15:20:33.917669 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.917622 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.935426 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.918123 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 15:20:33.962052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.944142 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.963435 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.962301 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 15:20:33.964983 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.964959 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:20:34.037083 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:33.965491 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:20:34.052278 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.052251 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:20:34.140028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.103253 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 15:20:34.140028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.139969 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:20:34.161022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.160154 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.200346 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.200252 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.251018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.250275 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30682f86-a130-4716-bbf3-6fc4756c0798" path="/var/lib/kubelet/pods/30682f86-a130-4716-bbf3-6fc4756c0798/volumes" Nov 02 15:20:34.251018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.250788 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:34.251018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.250838 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:20:34.251018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.250872 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:20:34.263971 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.263945 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:20:34.272008 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.269172 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.272008 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.269354 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:20:34.300715 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.300687 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:20:34.330169 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.328289 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.330169 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.328632 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:20:34.334493 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.334472 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:20:34.355362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.355338 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:20:34.416367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.415941 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:20:34.420456 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.418120 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:20:34.434906 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.434883 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:20:34.445767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.445749 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:20:34.451625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.451569 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:20:34.806611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.806521 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:20:34.807941 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:34.807910 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 15:20:35.787826 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:35.787786 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:20:35.788234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:35.787858 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 02 15:20:35.881755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:35.881722 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f"] Nov 02 15:20:35.882763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:35.882742 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b41f318d-c9ac-4d0f-8da2-9fac8931e0ea" containerName="revoke-keys" Nov 02 15:20:35.882837 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:35.882767 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41f318d-c9ac-4d0f-8da2-9fac8931e0ea" containerName="revoke-keys" Nov 02 15:20:35.882954 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:35.882940 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41f318d-c9ac-4d0f-8da2-9fac8931e0ea" containerName="revoke-keys" Nov 02 15:20:35.887256 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:35.887236 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:35.942545 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:35.942488 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:20:35.942769 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:35.942570 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:20:35.947475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:35.947446 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f"] Nov 02 15:20:36.061562 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.061473 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6189297-jw45f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:36.061828 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.061802 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bls\" (UniqueName: \"kubernetes.io/projected/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-kube-api-access-72bls\") pod \"maas-api-revoke-sub-d6189297-jw45f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:36.061901 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.061861 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6189297-jw45f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:36.163214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.163179 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bls\" (UniqueName: \"kubernetes.io/projected/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-kube-api-access-72bls\") pod \"maas-api-revoke-sub-d6189297-jw45f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:36.163400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.163226 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6189297-jw45f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:36.163400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.163337 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6189297-jw45f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:36.164089 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.164064 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6189297-jw45f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:36.165752 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.165727 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6189297-jw45f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:36.222590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.222561 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bls\" (UniqueName: \"kubernetes.io/projected/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-kube-api-access-72bls\") pod \"maas-api-revoke-sub-d6189297-jw45f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:36.505040 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.505007 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:36.891383 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.891356 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f"] Nov 02 15:20:36.896038 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:36.895982 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe793fd_1322_4e57_b2c0_29b2ba95c35f.slice/crio-363fe6ee0f3c1a9ace4be89ed8e7a3daafb39c2fa46a470d22946dbfa7dea926 WatchSource:0}: Error finding container 363fe6ee0f3c1a9ace4be89ed8e7a3daafb39c2fa46a470d22946dbfa7dea926: Status 404 returned error can't find the container with id 363fe6ee0f3c1a9ace4be89ed8e7a3daafb39c2fa46a470d22946dbfa7dea926 Nov 02 15:20:36.985650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.985609 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:36.986113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.985990 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:36.986811 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:36.986794 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:20:37.649329 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:37.649295 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" event={"ID":"cfe793fd-1322-4e57-b2c0-29b2ba95c35f","Type":"ContainerStarted","Data":"ae95e83d20ec95ed11f41b6d797f8c2d0b47ea4c24524fbbeaabfd45e2a4c073"} Nov 02 15:20:37.649535 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:37.649340 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" event={"ID":"cfe793fd-1322-4e57-b2c0-29b2ba95c35f","Type":"ContainerStarted","Data":"363fe6ee0f3c1a9ace4be89ed8e7a3daafb39c2fa46a470d22946dbfa7dea926"} Nov 02 15:20:37.766014 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:37.764136 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" podStartSLOduration=2.764109035 podStartE2EDuration="2.764109035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:37.763078222 +0000 UTC m=+1339.965956283" watchObservedRunningTime="2026-11-02 15:20:37.764109035 +0000 UTC m=+1339.966987096" Nov 02 15:20:38.657989 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:38.657951 5476 generic.go:356] "Generic (PLEG): container finished" podID="cfe793fd-1322-4e57-b2c0-29b2ba95c35f" containerID="ae95e83d20ec95ed11f41b6d797f8c2d0b47ea4c24524fbbeaabfd45e2a4c073" exitCode=0 Nov 02 15:20:38.658403 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:38.658038 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" event={"ID":"cfe793fd-1322-4e57-b2c0-29b2ba95c35f","Type":"ContainerDied","Data":"ae95e83d20ec95ed11f41b6d797f8c2d0b47ea4c24524fbbeaabfd45e2a4c073"} Nov 02 15:20:39.815752 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:39.815727 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:39.912601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:39.912575 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-xks-ca\") pod \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " Nov 02 15:20:39.912737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:39.912633 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-openshift-ca\") pod \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " Nov 02 15:20:39.912737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:39.912713 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-kube-api-access-72bls\" (UniqueName: \"kubernetes.io/projected/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-kube-api-access-72bls\") pod \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\" (UID: \"cfe793fd-1322-4e57-b2c0-29b2ba95c35f\") " Nov 02 15:20:39.913115 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:39.913078 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-openshift-ca" pod "cfe793fd-1322-4e57-b2c0-29b2ba95c35f" (UID: "cfe793fd-1322-4e57-b2c0-29b2ba95c35f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:20:39.913240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:39.913222 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:39.914719 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:39.914693 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-xks-ca" pod "cfe793fd-1322-4e57-b2c0-29b2ba95c35f" (UID: "cfe793fd-1322-4e57-b2c0-29b2ba95c35f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:20:39.914814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:39.914760 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-kube-api-access-72bls" pod "cfe793fd-1322-4e57-b2c0-29b2ba95c35f" (UID: "cfe793fd-1322-4e57-b2c0-29b2ba95c35f"). InnerVolumeSpecName "kube-api-access-72bls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:20:40.014658 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:40.014631 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:40.014658 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:40.014657 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72bls\" (UniqueName: \"kubernetes.io/projected/cfe793fd-1322-4e57-b2c0-29b2ba95c35f-kube-api-access-72bls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:40.675240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:40.675210 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" event={"ID":"cfe793fd-1322-4e57-b2c0-29b2ba95c35f","Type":"ContainerDied","Data":"363fe6ee0f3c1a9ace4be89ed8e7a3daafb39c2fa46a470d22946dbfa7dea926"} Nov 02 15:20:40.675437 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:40.675256 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="363fe6ee0f3c1a9ace4be89ed8e7a3daafb39c2fa46a470d22946dbfa7dea926" Nov 02 15:20:40.675437 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:40.675256 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f" Nov 02 15:20:43.016764 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.016728 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg"] Nov 02 15:20:43.017532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.017514 5476 cpu_manager.go:414] "containerMap: removing container" podUID="cfe793fd-1322-4e57-b2c0-29b2ba95c35f" containerName="revoke-keys" Nov 02 15:20:43.017584 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.017535 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe793fd-1322-4e57-b2c0-29b2ba95c35f" containerName="revoke-keys" Nov 02 15:20:43.017758 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.017744 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe793fd-1322-4e57-b2c0-29b2ba95c35f" containerName="revoke-keys" Nov 02 15:20:43.021673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.021656 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.050921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.050893 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.051047 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.050927 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb515d1-ae20-4c42-bba2-a719b3049927-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.051157 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.051137 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.051219 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.051183 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.051314 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.051279 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.051379 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.051355 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.051615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.051380 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.051615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.051518 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.051615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.051564 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grsrm\" (UniqueName: \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-kube-api-access-grsrm\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.134217 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.134192 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg"] Nov 02 15:20:43.152615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.152591 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.152748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.152635 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.152748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.152668 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.152748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.152719 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.152748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.152744 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.152957 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.152845 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.152957 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.152889 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grsrm\" (UniqueName: \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-kube-api-access-grsrm\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.152957 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.152946 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.153119 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.152976 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb515d1-ae20-4c42-bba2-a719b3049927-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.153119 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.153015 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.153218 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.153166 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.153268 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.153241 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.153353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.153335 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.153653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.153629 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb515d1-ae20-4c42-bba2-a719b3049927-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.155204 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.155176 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.155420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.155400 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.217583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.217558 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.218097 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.218080 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grsrm\" (UniqueName: \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-kube-api-access-grsrm\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.337114 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.337043 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:43.782584 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.782560 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg"] Nov 02 15:20:43.784960 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:43.784935 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb515d1_ae20_4c42_bba2_a719b3049927.slice/crio-dac87d1ad0cbfee0ed64366c9c9178530319ce92c76f5140719e419c0c2fb247 WatchSource:0}: Error finding container dac87d1ad0cbfee0ed64366c9c9178530319ce92c76f5140719e419c0c2fb247: Status 404 returned error can't find the container with id dac87d1ad0cbfee0ed64366c9c9178530319ce92c76f5140719e419c0c2fb247 Nov 02 15:20:43.787149 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.787121 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:43.787211 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.787182 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:43.787254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:43.787226 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:44.155315 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.155257 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9"] Nov 02 15:20:44.159525 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.159505 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.226111 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.226070 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9"] Nov 02 15:20:44.272592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.272550 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.272795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.272625 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.272795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.272718 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.272795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.272754 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.272968 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.272800 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.272968 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.272827 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.272968 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.272880 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.272968 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.272915 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.273146 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.273050 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49sld\" (UniqueName: \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-kube-api-access-49sld\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.396693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.395900 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49sld\" (UniqueName: \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-kube-api-access-49sld\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.396693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.396154 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.396693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.396213 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.396693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.396300 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.396693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.396353 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.396693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.396406 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.396693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.396436 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.396693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.396483 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.396693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.396527 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.397368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.396783 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.397368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.397186 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.397368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.397352 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.397526 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.397389 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.397578 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.397541 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.399388 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.399363 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.399850 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.399827 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.457049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.455717 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49sld\" (UniqueName: \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-kube-api-access-49sld\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.457049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.455890 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.492317 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.492271 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:44.706242 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.706203 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" event={"ID":"1fb515d1-ae20-4c42-bba2-a719b3049927","Type":"ContainerStarted","Data":"2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f"} Nov 02 15:20:44.706403 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.706300 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" event={"ID":"1fb515d1-ae20-4c42-bba2-a719b3049927","Type":"ContainerStarted","Data":"dac87d1ad0cbfee0ed64366c9c9178530319ce92c76f5140719e419c0c2fb247"} Nov 02 15:20:44.823730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:44.823627 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" podStartSLOduration=2.823594419 podStartE2EDuration="2.823594419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:44.821655527 +0000 UTC m=+1347.024533587" watchObservedRunningTime="2026-11-02 15:20:44.823594419 +0000 UTC m=+1347.026472481" Nov 02 15:20:45.002297 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:45.002266 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9"] Nov 02 15:20:45.002534 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:45.002511 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71c0212_6bf1_43ed_9d89_9dddc4cdba4a.slice/crio-116e38d86b5bfdfee78d2aea2b4161eae2add216f973a4a8ca676486ce7f6d8f WatchSource:0}: Error finding container 116e38d86b5bfdfee78d2aea2b4161eae2add216f973a4a8ca676486ce7f6d8f: Status 404 returned error can't find the container with id 116e38d86b5bfdfee78d2aea2b4161eae2add216f973a4a8ca676486ce7f6d8f Nov 02 15:20:45.005240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:45.005209 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:45.005338 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:45.005280 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:45.005338 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:45.005323 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:45.337956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:45.337924 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:45.339828 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:45.339803 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:20:45.339961 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:45.339845 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" podUID="1fb515d1-ae20-4c42-bba2-a719b3049927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 15:20:45.730476 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:45.730432 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" event={"ID":"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a","Type":"ContainerStarted","Data":"9c3b0f25541ffecea7e8a236f2deb2ffb6642dd141c86d3012d25a0ea9a7c09f"} Nov 02 15:20:45.730673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:45.730651 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" event={"ID":"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a","Type":"ContainerStarted","Data":"116e38d86b5bfdfee78d2aea2b4161eae2add216f973a4a8ca676486ce7f6d8f"} Nov 02 15:20:45.850033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:45.849965 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" podStartSLOduration=1.8499522000000002 podStartE2EDuration="1.8499522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:45.848485701 +0000 UTC m=+1348.051363772" watchObservedRunningTime="2026-11-02 15:20:45.8499522 +0000 UTC m=+1348.052830258" Nov 02 15:20:46.337764 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:46.337728 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:20:46.337924 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:46.337789 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" podUID="1fb515d1-ae20-4c42-bba2-a719b3049927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 15:20:46.495019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:46.494420 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:46.516018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:46.510167 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:20:46.516018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:46.510253 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 15:20:47.337763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:47.337726 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:20:47.337977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:47.337782 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" podUID="1fb515d1-ae20-4c42-bba2-a719b3049927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 15:20:47.507019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:47.506358 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:20:47.507019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:47.506421 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 15:20:48.060973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:48.060941 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g"] Nov 02 15:20:48.128093 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:48.127314 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92c8200f-x6h4g"] Nov 02 15:20:48.338373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:48.338287 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:20:48.338373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:48.338352 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" podUID="1fb515d1-ae20-4c42-bba2-a719b3049927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 15:20:48.513458 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:48.513424 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:20:48.514064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:48.514030 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 15:20:49.337837 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.337783 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:20:49.338027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.337852 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" podUID="1fb515d1-ae20-4c42-bba2-a719b3049927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 15:20:49.358185 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.358152 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc"] Nov 02 15:20:49.362760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.362739 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:49.416382 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.416356 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:20:49.427361 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.427333 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc"] Nov 02 15:20:49.463251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.463222 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:20:49.499154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.499120 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:20:49.499451 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.499405 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 15:20:49.527436 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.527389 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnk9t\" (UniqueName: \"kubernetes.io/projected/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-kube-api-access-pnk9t\") pod \"maas-api-revoke-sub-fdc3ec49-c85qc\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:49.528156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.528131 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc3ec49-c85qc\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:49.528289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.528187 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc3ec49-c85qc\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:49.629607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.629571 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnk9t\" (UniqueName: \"kubernetes.io/projected/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-kube-api-access-pnk9t\") pod \"maas-api-revoke-sub-fdc3ec49-c85qc\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:49.629801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.629631 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc3ec49-c85qc\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:49.629984 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.629962 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc3ec49-c85qc\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:49.630239 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.630217 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc3ec49-c85qc\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:49.632349 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.632326 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc3ec49-c85qc\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:49.717046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.713744 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnk9t\" (UniqueName: \"kubernetes.io/projected/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-kube-api-access-pnk9t\") pod \"maas-api-revoke-sub-fdc3ec49-c85qc\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:49.717046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.714212 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:49.923884 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:49.923801 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="042e5e11-0753-4d6d-b606-e0a2a32db585" path="/var/lib/kubelet/pods/042e5e11-0753-4d6d-b606-e0a2a32db585/volumes" Nov 02 15:20:50.129030 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:50.128983 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc"] Nov 02 15:20:50.132496 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:50.132466 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bbacd3_5f49_4704_a20b_ef15c4cc904c.slice/crio-33c380c83352b7d98b15fa88340e1a002c495fe49f3ff1ce3e442b62e2620767 WatchSource:0}: Error finding container 33c380c83352b7d98b15fa88340e1a002c495fe49f3ff1ce3e442b62e2620767: Status 404 returned error can't find the container with id 33c380c83352b7d98b15fa88340e1a002c495fe49f3ff1ce3e442b62e2620767 Nov 02 15:20:50.149291 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:20:50.149260 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:20:50.338053 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:50.338022 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:20:50.338219 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:50.338073 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" podUID="1fb515d1-ae20-4c42-bba2-a719b3049927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 15:20:50.497090 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:50.496442 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:20:50.497090 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:50.496528 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 15:20:50.810416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:50.810320 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" event={"ID":"a5bbacd3-5f49-4704-a20b-ef15c4cc904c","Type":"ContainerStarted","Data":"ceeac6a8847a1cc1c28c7efca3d148ac63e57feb32b51f2a8b43fece632d9725"} Nov 02 15:20:50.810416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:50.810396 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" event={"ID":"a5bbacd3-5f49-4704-a20b-ef15c4cc904c","Type":"ContainerStarted","Data":"33c380c83352b7d98b15fa88340e1a002c495fe49f3ff1ce3e442b62e2620767"} Nov 02 15:20:50.925319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:50.925251 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" podStartSLOduration=1.9252324029999999 podStartE2EDuration="1.925232403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:50.921542769 +0000 UTC m=+1353.124420829" watchObservedRunningTime="2026-11-02 15:20:50.925232403 +0000 UTC m=+1353.128110463" Nov 02 15:20:51.342776 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:51.342743 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:51.343125 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:51.343105 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:51.343851 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:51.343828 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:20:51.494690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:51.494602 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:20:51.494690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:51.494660 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 15:20:51.820503 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:51.820411 5476 generic.go:356] "Generic (PLEG): container finished" podID="a5bbacd3-5f49-4704-a20b-ef15c4cc904c" containerID="ceeac6a8847a1cc1c28c7efca3d148ac63e57feb32b51f2a8b43fece632d9725" exitCode=0 Nov 02 15:20:51.820975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:51.820495 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" event={"ID":"a5bbacd3-5f49-4704-a20b-ef15c4cc904c","Type":"ContainerDied","Data":"ceeac6a8847a1cc1c28c7efca3d148ac63e57feb32b51f2a8b43fece632d9725"} Nov 02 15:20:52.499045 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:52.499008 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:52.499500 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:52.499482 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:52.500187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:52.500164 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:20:53.000118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.000090 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:53.077450 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.077423 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-openshift-ca\") pod \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " Nov 02 15:20:53.077579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.077488 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-kube-api-access-pnk9t\" (UniqueName: \"kubernetes.io/projected/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-kube-api-access-pnk9t\") pod \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " Nov 02 15:20:53.077579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.077533 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-xks-ca\") pod \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\" (UID: \"a5bbacd3-5f49-4704-a20b-ef15c4cc904c\") " Nov 02 15:20:53.077756 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.077736 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-openshift-ca" pod "a5bbacd3-5f49-4704-a20b-ef15c4cc904c" (UID: "a5bbacd3-5f49-4704-a20b-ef15c4cc904c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:20:53.077985 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.077958 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:53.079546 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.079487 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-xks-ca" pod "a5bbacd3-5f49-4704-a20b-ef15c4cc904c" (UID: "a5bbacd3-5f49-4704-a20b-ef15c4cc904c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:20:53.079663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.079643 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-kube-api-access-pnk9t" pod "a5bbacd3-5f49-4704-a20b-ef15c4cc904c" (UID: "a5bbacd3-5f49-4704-a20b-ef15c4cc904c"). InnerVolumeSpecName "kube-api-access-pnk9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:20:53.179115 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.179074 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnk9t\" (UniqueName: \"kubernetes.io/projected/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-kube-api-access-pnk9t\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:53.179115 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.179113 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5bbacd3-5f49-4704-a20b-ef15c4cc904c-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:53.837518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.837491 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" Nov 02 15:20:53.837679 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.837531 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc" event={"ID":"a5bbacd3-5f49-4704-a20b-ef15c4cc904c","Type":"ContainerDied","Data":"33c380c83352b7d98b15fa88340e1a002c495fe49f3ff1ce3e442b62e2620767"} Nov 02 15:20:53.837679 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:53.837583 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33c380c83352b7d98b15fa88340e1a002c495fe49f3ff1ce3e442b62e2620767" Nov 02 15:20:54.488101 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.488069 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c"] Nov 02 15:20:54.488643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.488630 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a5bbacd3-5f49-4704-a20b-ef15c4cc904c" containerName="revoke-keys" Nov 02 15:20:54.488680 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.488644 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5bbacd3-5f49-4704-a20b-ef15c4cc904c" containerName="revoke-keys" Nov 02 15:20:54.488800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.488791 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5bbacd3-5f49-4704-a20b-ef15c4cc904c" containerName="revoke-keys" Nov 02 15:20:54.492522 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.492506 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:54.496904 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.496879 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a946b8c7-ths4c\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:54.497015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.496933 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq2t9\" (UniqueName: \"kubernetes.io/projected/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-kube-api-access-hq2t9\") pod \"maas-api-revoke-sub-a946b8c7-ths4c\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:54.497083 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.497070 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a946b8c7-ths4c\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:54.542906 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.542882 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:20:54.589211 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.589189 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:20:54.594373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.594355 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c"] Nov 02 15:20:54.598079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.598062 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a946b8c7-ths4c\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:54.598133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.598098 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq2t9\" (UniqueName: \"kubernetes.io/projected/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-kube-api-access-hq2t9\") pod \"maas-api-revoke-sub-a946b8c7-ths4c\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:54.598174 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.598165 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a946b8c7-ths4c\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:54.598737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.598717 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a946b8c7-ths4c\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:54.600383 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.600363 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a946b8c7-ths4c\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:54.656038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.655979 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq2t9\" (UniqueName: \"kubernetes.io/projected/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-kube-api-access-hq2t9\") pod \"maas-api-revoke-sub-a946b8c7-ths4c\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:54.805740 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:54.805678 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:55.051876 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:55.051843 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm"] Nov 02 15:20:55.057242 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:55.057176 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b15ac65d-hpbqm"] Nov 02 15:20:55.183394 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:55.183297 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c"] Nov 02 15:20:55.186416 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:55.186389 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b2bb10_1a41_453e_b6f3_fecfcd0e613d.slice/crio-047827d9ea8bac505741a04dd8ff85b8375815d9b098dcfa37c1082761ff7682 WatchSource:0}: Error finding container 047827d9ea8bac505741a04dd8ff85b8375815d9b098dcfa37c1082761ff7682: Status 404 returned error can't find the container with id 047827d9ea8bac505741a04dd8ff85b8375815d9b098dcfa37c1082761ff7682 Nov 02 15:20:55.858782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:55.858740 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" event={"ID":"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d","Type":"ContainerStarted","Data":"f561494bd9666258551ae6a8b25fc446a52742123f3693cf34c235e6c42dcb26"} Nov 02 15:20:55.859313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:55.858802 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" event={"ID":"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d","Type":"ContainerStarted","Data":"047827d9ea8bac505741a04dd8ff85b8375815d9b098dcfa37c1082761ff7682"} Nov 02 15:20:55.922774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:55.922746 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37d2250-b5b6-4807-bd08-cd96c1424e21" path="/var/lib/kubelet/pods/d37d2250-b5b6-4807-bd08-cd96c1424e21/volumes" Nov 02 15:20:55.973233 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:55.973184 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" podStartSLOduration=1.973169771 podStartE2EDuration="1.973169771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:55.969076519 +0000 UTC m=+1358.171954577" watchObservedRunningTime="2026-11-02 15:20:55.973169771 +0000 UTC m=+1358.176047830" Nov 02 15:20:56.868118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:56.868080 5476 generic.go:356] "Generic (PLEG): container finished" podID="a9b2bb10-1a41-453e-b6f3-fecfcd0e613d" containerID="f561494bd9666258551ae6a8b25fc446a52742123f3693cf34c235e6c42dcb26" exitCode=0 Nov 02 15:20:56.868546 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:56.868170 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" event={"ID":"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d","Type":"ContainerDied","Data":"f561494bd9666258551ae6a8b25fc446a52742123f3693cf34c235e6c42dcb26"} Nov 02 15:20:57.293198 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.293098 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96"] Nov 02 15:20:57.297282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.297266 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.332963 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.332939 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.333104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.332970 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.333104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.332987 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fe2e6c-e885-4e1d-b974-54a98225dad1-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.333218 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.333205 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.333257 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.333249 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.333325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.333306 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.333358 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.333342 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.333426 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.333411 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.333525 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.333512 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhfn\" (UniqueName: \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-kube-api-access-mnhfn\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.397722 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.397692 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9"] Nov 02 15:20:57.403085 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.403060 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.403336 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.403297 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96"] Nov 02 15:20:57.434902 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.434548 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.434902 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.434600 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.434902 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.434656 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.436161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.434973 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.436161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.435050 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.436161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.435399 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.436161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.435462 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1549618-289f-426e-997d-277d2705d0db-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.436161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436145 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.436432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436228 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.436432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436261 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l55h\" (UniqueName: \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-kube-api-access-7l55h\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.436432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436282 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.436432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436300 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.436432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436335 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.436432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436367 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhfn\" (UniqueName: \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-kube-api-access-mnhfn\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.436432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436394 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.436432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436417 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1549618-289f-426e-997d-277d2705d0db-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.436432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436437 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.436890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436457 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.436890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436472 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fe2e6c-e885-4e1d-b974-54a98225dad1-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.436890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.436490 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.437131 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.437109 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.437288 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.437259 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.438089 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.438065 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fe2e6c-e885-4e1d-b974-54a98225dad1-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.445842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.445797 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.449482 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.449428 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9"] Nov 02 15:20:57.451506 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.451485 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.495953 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.495920 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhfn\" (UniqueName: \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-kube-api-access-mnhfn\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.496615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.496586 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.537800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.537764 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1549618-289f-426e-997d-277d2705d0db-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.537800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.537809 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.537954 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.537835 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l55h\" (UniqueName: \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-kube-api-access-7l55h\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.537954 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.537862 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.537954 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.537910 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.537954 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.537944 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1549618-289f-426e-997d-277d2705d0db-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.538167 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.538058 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.538259 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.538210 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.538397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.538289 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.538397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.538321 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.538397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.538332 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.538397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.538381 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.538609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.538569 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.538766 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.538737 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1549618-289f-426e-997d-277d2705d0db-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.540276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.540256 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1549618-289f-426e-997d-277d2705d0db-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.540358 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.540279 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.597126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.597070 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l55h\" (UniqueName: \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-kube-api-access-7l55h\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.598103 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.598087 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:57.613702 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.613681 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:57.718713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:57.718680 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:58.010612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.010589 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:58.045825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.045798 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96"] Nov 02 15:20:58.046135 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.046116 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-xks-ca\") pod \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " Nov 02 15:20:58.046249 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.046235 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-kube-api-access-hq2t9\" (UniqueName: \"kubernetes.io/projected/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-kube-api-access-hq2t9\") pod \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " Nov 02 15:20:58.046388 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.046364 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-openshift-ca\") pod \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\" (UID: \"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d\") " Nov 02 15:20:58.046779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.046751 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-openshift-ca" pod "a9b2bb10-1a41-453e-b6f3-fecfcd0e613d" (UID: "a9b2bb10-1a41-453e-b6f3-fecfcd0e613d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:20:58.047144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.047120 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:58.048231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.048211 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-xks-ca" pod "a9b2bb10-1a41-453e-b6f3-fecfcd0e613d" (UID: "a9b2bb10-1a41-453e-b6f3-fecfcd0e613d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:20:58.048373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.048352 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-kube-api-access-hq2t9" pod "a9b2bb10-1a41-453e-b6f3-fecfcd0e613d" (UID: "a9b2bb10-1a41-453e-b6f3-fecfcd0e613d"). InnerVolumeSpecName "kube-api-access-hq2t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:20:58.090217 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:58.090168 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fe2e6c_e885_4e1d_b974_54a98225dad1.slice/crio-d838255548c8299f2fcff0de9ddb82884f14903d5826c7dc1ca841378c156cd4 WatchSource:0}: Error finding container d838255548c8299f2fcff0de9ddb82884f14903d5826c7dc1ca841378c156cd4: Status 404 returned error can't find the container with id d838255548c8299f2fcff0de9ddb82884f14903d5826c7dc1ca841378c156cd4 Nov 02 15:20:58.092848 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.092818 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.092927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.092886 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.092927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.092917 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.106403 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.106379 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9"] Nov 02 15:20:58.109058 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:20:58.109009 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1549618_289f_426e_997d_277d2705d0db.slice/crio-8e3da0505bf6cba760dc64221a8cbfe96dc892cd13b0a1cd55acde7bb9b161c2 WatchSource:0}: Error finding container 8e3da0505bf6cba760dc64221a8cbfe96dc892cd13b0a1cd55acde7bb9b161c2: Status 404 returned error can't find the container with id 8e3da0505bf6cba760dc64221a8cbfe96dc892cd13b0a1cd55acde7bb9b161c2 Nov 02 15:20:58.111330 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.111301 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.111396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.111370 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.111454 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.111398 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.148715 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.148559 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq2t9\" (UniqueName: \"kubernetes.io/projected/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-kube-api-access-hq2t9\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:58.148715 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.148587 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:20:58.924402 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.924358 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" event={"ID":"80fe2e6c-e885-4e1d-b974-54a98225dad1","Type":"ContainerStarted","Data":"3bb713b61af2b2d6c3803c3ae456ac6d5bdae6eb75d4b5c4ca56aa49d8bc4b74"} Nov 02 15:20:58.924633 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.924427 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" event={"ID":"80fe2e6c-e885-4e1d-b974-54a98225dad1","Type":"ContainerStarted","Data":"d838255548c8299f2fcff0de9ddb82884f14903d5826c7dc1ca841378c156cd4"} Nov 02 15:20:58.927077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.927041 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" event={"ID":"a1549618-289f-426e-997d-277d2705d0db","Type":"ContainerStarted","Data":"20f794160c7907d364c14cff8912f5f965d816f8fab7ca05823af7a57a32ad64"} Nov 02 15:20:58.927197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.927098 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" event={"ID":"a1549618-289f-426e-997d-277d2705d0db","Type":"ContainerStarted","Data":"8e3da0505bf6cba760dc64221a8cbfe96dc892cd13b0a1cd55acde7bb9b161c2"} Nov 02 15:20:58.928739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.928713 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" event={"ID":"a9b2bb10-1a41-453e-b6f3-fecfcd0e613d","Type":"ContainerDied","Data":"047827d9ea8bac505741a04dd8ff85b8375815d9b098dcfa37c1082761ff7682"} Nov 02 15:20:58.928739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.928738 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c" Nov 02 15:20:58.928880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:58.928757 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="047827d9ea8bac505741a04dd8ff85b8375815d9b098dcfa37c1082761ff7682" Nov 02 15:20:59.060976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:59.060626 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" podStartSLOduration=2.060612416 podStartE2EDuration="2.060612416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:59.059482808 +0000 UTC m=+1361.262360851" watchObservedRunningTime="2026-11-02 15:20:59.060612416 +0000 UTC m=+1361.263490531" Nov 02 15:20:59.200808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:59.200670 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" podStartSLOduration=2.20062834 podStartE2EDuration="2.20062834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:59.175497357 +0000 UTC m=+1361.378375442" watchObservedRunningTime="2026-11-02 15:20:59.20062834 +0000 UTC m=+1361.403506401" Nov 02 15:20:59.621396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:59.621354 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:20:59.624294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:59.624267 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 15:20:59.624432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:59.624319 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 15:20:59.721020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:59.720343 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:20:59.722403 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:59.722261 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:20:59.722403 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:20:59.722328 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" podUID="a1549618-289f-426e-997d-277d2705d0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:21:00.621043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:00.615723 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 15:21:00.621043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:00.615785 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 15:21:00.721021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:00.719427 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:21:00.721021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:00.719487 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" podUID="a1549618-289f-426e-997d-277d2705d0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:21:01.616019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:01.615243 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 15:21:01.616019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:01.615308 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 15:21:01.725032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:01.723977 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:21:01.725032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:01.724072 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" podUID="a1549618-289f-426e-997d-277d2705d0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:21:02.621476 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:02.620973 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 15:21:02.621476 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:02.621082 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 15:21:02.721048 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:02.719454 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:21:02.721048 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:02.719530 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" podUID="a1549618-289f-426e-997d-277d2705d0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:21:03.616019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:03.615355 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 15:21:03.616019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:03.615418 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 15:21:03.720021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:03.719347 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:21:03.720021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:03.719407 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" podUID="a1549618-289f-426e-997d-277d2705d0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:21:04.614738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:04.614702 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 15:21:04.614941 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:04.614763 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 15:21:04.719579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:04.719543 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 15:21:04.719940 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:04.719618 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" podUID="a1549618-289f-426e-997d-277d2705d0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 15:21:05.636594 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.636563 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:21:05.636988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.636963 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:21:05.637929 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.637904 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:21:05.724079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.724051 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:21:05.724454 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.724319 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:21:05.724934 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.724917 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:21:05.736075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.736040 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb"] Nov 02 15:21:05.736638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.736623 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a9b2bb10-1a41-453e-b6f3-fecfcd0e613d" containerName="revoke-keys" Nov 02 15:21:05.736690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.736639 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b2bb10-1a41-453e-b6f3-fecfcd0e613d" containerName="revoke-keys" Nov 02 15:21:05.736744 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.736736 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b2bb10-1a41-453e-b6f3-fecfcd0e613d" containerName="revoke-keys" Nov 02 15:21:05.740277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.740261 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:05.754674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.754653 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb"] Nov 02 15:21:05.789152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.789126 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:21:05.789290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.789201 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:21:05.797621 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.797589 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qd75\" (UniqueName: \"kubernetes.io/projected/ad663c1c-ec73-4f47-b9ca-55d519911296-kube-api-access-2qd75\") pod \"maas-api-revoke-sub-57f8f74c-h9wnb\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:05.797738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.797663 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f8f74c-h9wnb\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:05.797811 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.797787 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f8f74c-h9wnb\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:05.899617 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.899547 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qd75\" (UniqueName: \"kubernetes.io/projected/ad663c1c-ec73-4f47-b9ca-55d519911296-kube-api-access-2qd75\") pod \"maas-api-revoke-sub-57f8f74c-h9wnb\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:05.899617 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.899588 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f8f74c-h9wnb\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:05.899776 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.899639 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f8f74c-h9wnb\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:05.900277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.900256 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f8f74c-h9wnb\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:05.901907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.901887 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f8f74c-h9wnb\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:05.965888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:05.965854 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qd75\" (UniqueName: \"kubernetes.io/projected/ad663c1c-ec73-4f47-b9ca-55d519911296-kube-api-access-2qd75\") pod \"maas-api-revoke-sub-57f8f74c-h9wnb\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:06.055178 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:06.055142 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:06.067101 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:06.067070 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj"] Nov 02 15:21:06.070523 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:06.070496 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba90c25-4lfrj"] Nov 02 15:21:06.458196 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:06.458069 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb"] Nov 02 15:21:06.461305 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:06.461275 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad663c1c_ec73_4f47_b9ca_55d519911296.slice/crio-48dc86fd9f720d6770156e9594705b10d851aae64508c9b1a531d708fd5adede WatchSource:0}: Error finding container 48dc86fd9f720d6770156e9594705b10d851aae64508c9b1a531d708fd5adede: Status 404 returned error can't find the container with id 48dc86fd9f720d6770156e9594705b10d851aae64508c9b1a531d708fd5adede Nov 02 15:21:06.997462 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:06.997422 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" event={"ID":"ad663c1c-ec73-4f47-b9ca-55d519911296","Type":"ContainerStarted","Data":"573f2e023ba8f4cca2d7105a66c4a48838157c0083fb9819432277e2a3a7280c"} Nov 02 15:21:06.997853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:06.997493 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" event={"ID":"ad663c1c-ec73-4f47-b9ca-55d519911296","Type":"ContainerStarted","Data":"48dc86fd9f720d6770156e9594705b10d851aae64508c9b1a531d708fd5adede"} Nov 02 15:21:07.106511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:07.106439 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" podStartSLOduration=2.106426651 podStartE2EDuration="2.106426651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:07.106042592 +0000 UTC m=+1369.308920653" watchObservedRunningTime="2026-11-02 15:21:07.106426651 +0000 UTC m=+1369.309304710" Nov 02 15:21:07.409707 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:07.409635 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:21:07.960952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:07.960913 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c6dda6-87a3-46e8-8a39-52d8a66d5d77" path="/var/lib/kubelet/pods/b0c6dda6-87a3-46e8-8a39-52d8a66d5d77/volumes" Nov 02 15:21:08.043013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:08.042638 5476 generic.go:356] "Generic (PLEG): container finished" podID="ad663c1c-ec73-4f47-b9ca-55d519911296" containerID="573f2e023ba8f4cca2d7105a66c4a48838157c0083fb9819432277e2a3a7280c" exitCode=0 Nov 02 15:21:08.043013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:08.042835 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" event={"ID":"ad663c1c-ec73-4f47-b9ca-55d519911296","Type":"ContainerDied","Data":"573f2e023ba8f4cca2d7105a66c4a48838157c0083fb9819432277e2a3a7280c"} Nov 02 15:21:09.524077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:09.523088 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.006292 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.005626 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.337077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.333641 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 02 15:21:10.337077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.333929 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:21:10.408019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.377721 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.408019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.406558 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:21:10.433093 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.293319 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:21:10.457018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.450145 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.483018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.450598 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.483018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.450627 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 15:21:10.507014 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.450655 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.507014 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.458059 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.521171 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.495983 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf"] Nov 02 15:21:10.537026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.523776 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb9fc4f7-x5bwf"] Nov 02 15:21:10.564185 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.564147 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:21:10.687309 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.687269 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:10.706930 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.706889 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-xks-ca\") pod \"ad663c1c-ec73-4f47-b9ca-55d519911296\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " Nov 02 15:21:10.707737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.707711 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-openshift-ca\") pod \"ad663c1c-ec73-4f47-b9ca-55d519911296\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " Nov 02 15:21:10.717619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.707972 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad663c1c-ec73-4f47-b9ca-55d519911296-kube-api-access-2qd75\" (UniqueName: \"kubernetes.io/projected/ad663c1c-ec73-4f47-b9ca-55d519911296-kube-api-access-2qd75\") pod \"ad663c1c-ec73-4f47-b9ca-55d519911296\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " Nov 02 15:21:10.718315 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.712943 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-openshift-ca" pod "ad663c1c-ec73-4f47-b9ca-55d519911296" (UID: "ad663c1c-ec73-4f47-b9ca-55d519911296"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:10.718582 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.718524 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:10.973077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:10.964247 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-xks-ca" pod "ad663c1c-ec73-4f47-b9ca-55d519911296" (UID: "ad663c1c-ec73-4f47-b9ca-55d519911296"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:11.077053 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.064605 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" podUID="6abac5be-31c8-4f53-a556-cf43374c83a9" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.077053 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.065473 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad663c1c-ec73-4f47-b9ca-55d519911296-kube-api-access-2qd75" pod "ad663c1c-ec73-4f47-b9ca-55d519911296" (UID: "ad663c1c-ec73-4f47-b9ca-55d519911296"). InnerVolumeSpecName "kube-api-access-2qd75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:11.134020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.129043 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad663c1c-ec73-4f47-b9ca-55d519911296-kube-api-access-2qd75\" (UniqueName: \"kubernetes.io/projected/ad663c1c-ec73-4f47-b9ca-55d519911296-kube-api-access-2qd75\") pod \"ad663c1c-ec73-4f47-b9ca-55d519911296\" (UID: \"ad663c1c-ec73-4f47-b9ca-55d519911296\") " Nov 02 15:21:11.134020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.131325 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad663c1c-ec73-4f47-b9ca-55d519911296-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:11.134020 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:11.131564 5476 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/ad663c1c-ec73-4f47-b9ca-55d519911296/volumes/kubernetes.io~projected/kube-api-access-2qd75 Nov 02 15:21:11.134020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.131624 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad663c1c-ec73-4f47-b9ca-55d519911296-kube-api-access-2qd75" pod "ad663c1c-ec73-4f47-b9ca-55d519911296" (UID: "ad663c1c-ec73-4f47-b9ca-55d519911296"). InnerVolumeSpecName "kube-api-access-2qd75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:11.269870 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.269778 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qd75\" (UniqueName: \"kubernetes.io/projected/ad663c1c-ec73-4f47-b9ca-55d519911296-kube-api-access-2qd75\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:11.344480 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.344429 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.345426 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.345200 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 02 15:21:11.375022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.345864 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:21:11.383030 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.375569 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:21:11.404553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.404514 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.405186 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.405153 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:21:11.421030 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.420111 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.421030 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.420579 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:21:11.468018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.435776 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.468018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.436248 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:21:11.483025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.479924 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 15:21:11.515046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.495080 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.515046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.495643 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:21:11.515046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.509599 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:21:11.515046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.510516 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 15:21:11.515046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.511435 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:21:11.515046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.511833 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:21:11.574130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.561704 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 15:21:11.574130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.564291 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:21:11.574130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.564375 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:21:11.594130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.594072 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.811637 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:11.808831 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:21:12.320964 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.320936 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" Nov 02 15:21:12.348548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.348507 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:12.351244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.349531 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c" path="/var/lib/kubelet/pods/55c1dfed-5ec1-4d9c-ab74-8c3700fe4d4c/volumes" Nov 02 15:21:12.363677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.361477 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:21:12.399412 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.399375 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:21:12.399758 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.399737 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:21:12.402080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.401470 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb" event={"ID":"ad663c1c-ec73-4f47-b9ca-55d519911296","Type":"ContainerDied","Data":"48dc86fd9f720d6770156e9594705b10d851aae64508c9b1a531d708fd5adede"} Nov 02 15:21:12.402080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.401516 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48dc86fd9f720d6770156e9594705b10d851aae64508c9b1a531d708fd5adede" Nov 02 15:21:12.410951 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.410538 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:12.442017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.441298 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:12.455010 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.450893 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:21:12.455010 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.452980 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:21:12.490626 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.488243 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:21:12.490626 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.488965 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:21:12.490626 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.490322 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:21:12.492067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.492031 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:21:12.492202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.492087 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:21:12.495263 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:12.494845 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:21:13.973931 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:13.973896 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr"] Nov 02 15:21:13.975510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:13.975481 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ad663c1c-ec73-4f47-b9ca-55d519911296" containerName="revoke-keys" Nov 02 15:21:13.975510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:13.975508 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad663c1c-ec73-4f47-b9ca-55d519911296" containerName="revoke-keys" Nov 02 15:21:13.975737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:13.975721 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad663c1c-ec73-4f47-b9ca-55d519911296" containerName="revoke-keys" Nov 02 15:21:13.981678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:13.981658 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:14.032738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.032700 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr"] Nov 02 15:21:14.033742 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.033590 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:21:14.033859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.033771 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:21:14.101262 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.101222 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ccbb111-46jkr\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:14.101475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.101437 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ccbb111-46jkr\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:14.101592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.101573 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmws\" (UniqueName: \"kubernetes.io/projected/5afb5e83-9181-464a-bf1a-81a1a42d43ee-kube-api-access-qmmws\") pod \"maas-api-revoke-sub-4ccbb111-46jkr\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:14.203370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.203334 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ccbb111-46jkr\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:14.203536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.203426 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmws\" (UniqueName: \"kubernetes.io/projected/5afb5e83-9181-464a-bf1a-81a1a42d43ee-kube-api-access-qmmws\") pod \"maas-api-revoke-sub-4ccbb111-46jkr\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:14.203536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.203516 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ccbb111-46jkr\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:14.204051 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.204027 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ccbb111-46jkr\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:14.205935 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.205912 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ccbb111-46jkr\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:14.264519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.264451 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmws\" (UniqueName: \"kubernetes.io/projected/5afb5e83-9181-464a-bf1a-81a1a42d43ee-kube-api-access-qmmws\") pod \"maas-api-revoke-sub-4ccbb111-46jkr\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:14.308586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.308554 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:14.729180 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:14.729144 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr"] Nov 02 15:21:14.732891 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:14.732865 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afb5e83_9181_464a_bf1a_81a1a42d43ee.slice/crio-6b1532f94fd81ccb254af82d257e3c6f6f39065ecb5211318c387199e7758b3a WatchSource:0}: Error finding container 6b1532f94fd81ccb254af82d257e3c6f6f39065ecb5211318c387199e7758b3a: Status 404 returned error can't find the container with id 6b1532f94fd81ccb254af82d257e3c6f6f39065ecb5211318c387199e7758b3a Nov 02 15:21:15.378891 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:15.378856 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" event={"ID":"5afb5e83-9181-464a-bf1a-81a1a42d43ee","Type":"ContainerStarted","Data":"d5387157f43eba1d666743eb7fb98183358f19c9e3856d8b6527916e63a87703"} Nov 02 15:21:15.379286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:15.378918 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" event={"ID":"5afb5e83-9181-464a-bf1a-81a1a42d43ee","Type":"ContainerStarted","Data":"6b1532f94fd81ccb254af82d257e3c6f6f39065ecb5211318c387199e7758b3a"} Nov 02 15:21:15.495016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:15.493221 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" podStartSLOduration=2.493202127 podStartE2EDuration="2.493202127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:15.489072391 +0000 UTC m=+1377.691950455" watchObservedRunningTime="2026-11-02 15:21:15.493202127 +0000 UTC m=+1377.696080193" Nov 02 15:21:16.388494 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:16.388456 5476 generic.go:356] "Generic (PLEG): container finished" podID="5afb5e83-9181-464a-bf1a-81a1a42d43ee" containerID="d5387157f43eba1d666743eb7fb98183358f19c9e3856d8b6527916e63a87703" exitCode=0 Nov 02 15:21:16.388877 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:16.388540 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" event={"ID":"5afb5e83-9181-464a-bf1a-81a1a42d43ee","Type":"ContainerDied","Data":"d5387157f43eba1d666743eb7fb98183358f19c9e3856d8b6527916e63a87703"} Nov 02 15:21:16.931054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:16.931013 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz"] Nov 02 15:21:16.936105 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:16.936082 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:16.985920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:16.985892 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz"] Nov 02 15:21:17.055666 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.055635 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk"] Nov 02 15:21:17.070605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.070579 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76nxw\" (UniqueName: \"kubernetes.io/projected/7f5f7022-ae6a-4ecb-a28e-489226637f0e-kube-api-access-76nxw\") pod \"maas-api-revoke-sub-2844d9c8-mtrlz\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:17.070712 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.070618 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2844d9c8-mtrlz\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:17.070810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.070796 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2844d9c8-mtrlz\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:17.093891 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.093866 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ec33e7e-xpmjk"] Nov 02 15:21:17.171704 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.171668 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76nxw\" (UniqueName: \"kubernetes.io/projected/7f5f7022-ae6a-4ecb-a28e-489226637f0e-kube-api-access-76nxw\") pod \"maas-api-revoke-sub-2844d9c8-mtrlz\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:17.171849 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.171724 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2844d9c8-mtrlz\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:17.171919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.171879 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2844d9c8-mtrlz\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:17.172392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.172371 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2844d9c8-mtrlz\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:17.174133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.174115 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2844d9c8-mtrlz\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:17.229960 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.229881 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76nxw\" (UniqueName: \"kubernetes.io/projected/7f5f7022-ae6a-4ecb-a28e-489226637f0e-kube-api-access-76nxw\") pod \"maas-api-revoke-sub-2844d9c8-mtrlz\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:17.251927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.251902 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:17.574229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.574199 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:17.651292 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.651258 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz"] Nov 02 15:21:17.655327 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:17.655302 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5f7022_ae6a_4ecb_a28e_489226637f0e.slice/crio-25af0ca1981dea50ec33d4238461bc4cb22d5035aae9ba3831edf4a0c0e86d52 WatchSource:0}: Error finding container 25af0ca1981dea50ec33d4238461bc4cb22d5035aae9ba3831edf4a0c0e86d52: Status 404 returned error can't find the container with id 25af0ca1981dea50ec33d4238461bc4cb22d5035aae9ba3831edf4a0c0e86d52 Nov 02 15:21:17.679568 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.679547 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-xks-ca\") pod \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " Nov 02 15:21:17.679672 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.679587 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-openshift-ca\") pod \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " Nov 02 15:21:17.679873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.679854 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5afb5e83-9181-464a-bf1a-81a1a42d43ee-kube-api-access-qmmws\" (UniqueName: \"kubernetes.io/projected/5afb5e83-9181-464a-bf1a-81a1a42d43ee-kube-api-access-qmmws\") pod \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\" (UID: \"5afb5e83-9181-464a-bf1a-81a1a42d43ee\") " Nov 02 15:21:17.680012 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.679977 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-openshift-ca" pod "5afb5e83-9181-464a-bf1a-81a1a42d43ee" (UID: "5afb5e83-9181-464a-bf1a-81a1a42d43ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:17.680408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.680389 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:17.681461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.681434 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-xks-ca" pod "5afb5e83-9181-464a-bf1a-81a1a42d43ee" (UID: "5afb5e83-9181-464a-bf1a-81a1a42d43ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:17.681768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.681751 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afb5e83-9181-464a-bf1a-81a1a42d43ee-kube-api-access-qmmws" pod "5afb5e83-9181-464a-bf1a-81a1a42d43ee" (UID: "5afb5e83-9181-464a-bf1a-81a1a42d43ee"). InnerVolumeSpecName "kube-api-access-qmmws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:17.782287 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.782262 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5afb5e83-9181-464a-bf1a-81a1a42d43ee-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:17.782287 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.782285 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmmws\" (UniqueName: \"kubernetes.io/projected/5afb5e83-9181-464a-bf1a-81a1a42d43ee-kube-api-access-qmmws\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:17.924789 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:17.924760 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27b654d9-2735-4565-b4af-222947bbb2d2" path="/var/lib/kubelet/pods/27b654d9-2735-4565-b4af-222947bbb2d2/volumes" Nov 02 15:21:18.130706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:18.130678 5476 scope.go:122] "RemoveContainer" containerID="7ed1f1bf8b5f21716b59c7ae4bbd8019c0b8e70a0be9b04f2b29ff8d303c30d9" Nov 02 15:21:18.410242 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:18.410158 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" event={"ID":"5afb5e83-9181-464a-bf1a-81a1a42d43ee","Type":"ContainerDied","Data":"6b1532f94fd81ccb254af82d257e3c6f6f39065ecb5211318c387199e7758b3a"} Nov 02 15:21:18.410242 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:18.410181 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr" Nov 02 15:21:18.410242 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:18.410236 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b1532f94fd81ccb254af82d257e3c6f6f39065ecb5211318c387199e7758b3a" Nov 02 15:21:18.411780 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:18.411737 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" event={"ID":"7f5f7022-ae6a-4ecb-a28e-489226637f0e","Type":"ContainerStarted","Data":"323c8ed34701b24465a6e295079c8b36d7344461c3cb0379536a1e4fbe91474a"} Nov 02 15:21:18.411909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:18.411786 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" event={"ID":"7f5f7022-ae6a-4ecb-a28e-489226637f0e","Type":"ContainerStarted","Data":"25af0ca1981dea50ec33d4238461bc4cb22d5035aae9ba3831edf4a0c0e86d52"} Nov 02 15:21:18.524981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:18.524933 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" podStartSLOduration=2.524920533 podStartE2EDuration="2.524920533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:18.521103839 +0000 UTC m=+1380.723981900" watchObservedRunningTime="2026-11-02 15:21:18.524920533 +0000 UTC m=+1380.727798592" Nov 02 15:21:19.420574 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:19.420538 5476 generic.go:356] "Generic (PLEG): container finished" podID="7f5f7022-ae6a-4ecb-a28e-489226637f0e" containerID="323c8ed34701b24465a6e295079c8b36d7344461c3cb0379536a1e4fbe91474a" exitCode=0 Nov 02 15:21:19.421048 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:19.420620 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" event={"ID":"7f5f7022-ae6a-4ecb-a28e-489226637f0e","Type":"ContainerDied","Data":"323c8ed34701b24465a6e295079c8b36d7344461c3cb0379536a1e4fbe91474a"} Nov 02 15:21:19.944818 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:19.944782 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh"] Nov 02 15:21:19.945502 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:19.945482 5476 cpu_manager.go:414] "containerMap: removing container" podUID="5afb5e83-9181-464a-bf1a-81a1a42d43ee" containerName="revoke-keys" Nov 02 15:21:19.945502 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:19.945502 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5afb5e83-9181-464a-bf1a-81a1a42d43ee" containerName="revoke-keys" Nov 02 15:21:19.945684 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:19.945654 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5afb5e83-9181-464a-bf1a-81a1a42d43ee" containerName="revoke-keys" Nov 02 15:21:19.949378 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:19.949357 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:20.004817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.004783 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh"] Nov 02 15:21:20.011734 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.011701 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dphl9\" (UniqueName: \"kubernetes.io/projected/3b69522a-eaa1-4839-aa5e-221caa4cef34-kube-api-access-dphl9\") pod \"maas-api-revoke-sub-2ed2e6d3-2c5kh\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:20.012100 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.011975 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed2e6d3-2c5kh\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:20.012238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.012179 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed2e6d3-2c5kh\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:20.113785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.113746 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed2e6d3-2c5kh\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:20.113985 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.113902 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dphl9\" (UniqueName: \"kubernetes.io/projected/3b69522a-eaa1-4839-aa5e-221caa4cef34-kube-api-access-dphl9\") pod \"maas-api-revoke-sub-2ed2e6d3-2c5kh\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:20.113985 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.113939 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed2e6d3-2c5kh\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:20.114591 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.114567 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ed2e6d3-2c5kh\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:20.116233 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.116208 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ed2e6d3-2c5kh\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:20.146739 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:21:20.146705 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:21:20.171775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.171748 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dphl9\" (UniqueName: \"kubernetes.io/projected/3b69522a-eaa1-4839-aa5e-221caa4cef34-kube-api-access-dphl9\") pod \"maas-api-revoke-sub-2ed2e6d3-2c5kh\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:20.263625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.263546 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:20.603939 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.603912 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:20.642662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.642629 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh"] Nov 02 15:21:20.647089 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:20.647063 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b69522a_eaa1_4839_aa5e_221caa4cef34.slice/crio-8bb8e58208bcc9a4315f8b67d2fe9545a80d08c4649457b66ef9fe49f1729574 WatchSource:0}: Error finding container 8bb8e58208bcc9a4315f8b67d2fe9545a80d08c4649457b66ef9fe49f1729574: Status 404 returned error can't find the container with id 8bb8e58208bcc9a4315f8b67d2fe9545a80d08c4649457b66ef9fe49f1729574 Nov 02 15:21:20.724318 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.724291 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-openshift-ca\") pod \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " Nov 02 15:21:20.724442 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.724324 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-xks-ca\") pod \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " Nov 02 15:21:20.724442 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.724361 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5f7022-ae6a-4ecb-a28e-489226637f0e-kube-api-access-76nxw\" (UniqueName: \"kubernetes.io/projected/7f5f7022-ae6a-4ecb-a28e-489226637f0e-kube-api-access-76nxw\") pod \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\" (UID: \"7f5f7022-ae6a-4ecb-a28e-489226637f0e\") " Nov 02 15:21:20.724721 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.724689 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-openshift-ca" pod "7f5f7022-ae6a-4ecb-a28e-489226637f0e" (UID: "7f5f7022-ae6a-4ecb-a28e-489226637f0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:20.724884 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.724863 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:20.726527 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.726504 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5f7022-ae6a-4ecb-a28e-489226637f0e-kube-api-access-76nxw" pod "7f5f7022-ae6a-4ecb-a28e-489226637f0e" (UID: "7f5f7022-ae6a-4ecb-a28e-489226637f0e"). InnerVolumeSpecName "kube-api-access-76nxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:20.726642 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.726618 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-xks-ca" pod "7f5f7022-ae6a-4ecb-a28e-489226637f0e" (UID: "7f5f7022-ae6a-4ecb-a28e-489226637f0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:20.826003 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.825968 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5f7022-ae6a-4ecb-a28e-489226637f0e-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:20.826143 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:20.826014 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76nxw\" (UniqueName: \"kubernetes.io/projected/7f5f7022-ae6a-4ecb-a28e-489226637f0e-kube-api-access-76nxw\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:21.437937 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:21.437900 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" event={"ID":"3b69522a-eaa1-4839-aa5e-221caa4cef34","Type":"ContainerStarted","Data":"08afe708cf0a6b6dd2782bccb2666822a23eb0aec567960bfeb623b907916616"} Nov 02 15:21:21.438168 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:21.437955 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" event={"ID":"3b69522a-eaa1-4839-aa5e-221caa4cef34","Type":"ContainerStarted","Data":"8bb8e58208bcc9a4315f8b67d2fe9545a80d08c4649457b66ef9fe49f1729574"} Nov 02 15:21:21.439431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:21.439407 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" Nov 02 15:21:21.439556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:21.439435 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz" event={"ID":"7f5f7022-ae6a-4ecb-a28e-489226637f0e","Type":"ContainerDied","Data":"25af0ca1981dea50ec33d4238461bc4cb22d5035aae9ba3831edf4a0c0e86d52"} Nov 02 15:21:21.439556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:21.439480 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25af0ca1981dea50ec33d4238461bc4cb22d5035aae9ba3831edf4a0c0e86d52" Nov 02 15:21:21.551059 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:21.551013 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" podStartSLOduration=2.5509854130000003 podStartE2EDuration="2.550985413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:21.547613954 +0000 UTC m=+1383.750492014" watchObservedRunningTime="2026-11-02 15:21:21.550985413 +0000 UTC m=+1383.753863472" Nov 02 15:21:22.468035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:22.461044 5476 generic.go:356] "Generic (PLEG): container finished" podID="3b69522a-eaa1-4839-aa5e-221caa4cef34" containerID="08afe708cf0a6b6dd2782bccb2666822a23eb0aec567960bfeb623b907916616" exitCode=0 Nov 02 15:21:22.468035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:22.461156 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" event={"ID":"3b69522a-eaa1-4839-aa5e-221caa4cef34","Type":"ContainerDied","Data":"08afe708cf0a6b6dd2782bccb2666822a23eb0aec567960bfeb623b907916616"} Nov 02 15:21:23.050347 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.050314 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r"] Nov 02 15:21:23.054438 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.054418 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf0b8bb-8s76r"] Nov 02 15:21:23.644045 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.644021 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:23.767508 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.767410 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-xks-ca\") pod \"3b69522a-eaa1-4839-aa5e-221caa4cef34\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " Nov 02 15:21:23.767508 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.767464 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b69522a-eaa1-4839-aa5e-221caa4cef34-kube-api-access-dphl9\" (UniqueName: \"kubernetes.io/projected/3b69522a-eaa1-4839-aa5e-221caa4cef34-kube-api-access-dphl9\") pod \"3b69522a-eaa1-4839-aa5e-221caa4cef34\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " Nov 02 15:21:23.767777 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.767755 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-openshift-ca\") pod \"3b69522a-eaa1-4839-aa5e-221caa4cef34\" (UID: \"3b69522a-eaa1-4839-aa5e-221caa4cef34\") " Nov 02 15:21:23.768294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.768265 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-openshift-ca" pod "3b69522a-eaa1-4839-aa5e-221caa4cef34" (UID: "3b69522a-eaa1-4839-aa5e-221caa4cef34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:23.768414 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.768393 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:23.769774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.769750 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-xks-ca" pod "3b69522a-eaa1-4839-aa5e-221caa4cef34" (UID: "3b69522a-eaa1-4839-aa5e-221caa4cef34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:23.769910 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.769861 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b69522a-eaa1-4839-aa5e-221caa4cef34-kube-api-access-dphl9" pod "3b69522a-eaa1-4839-aa5e-221caa4cef34" (UID: "3b69522a-eaa1-4839-aa5e-221caa4cef34"). InnerVolumeSpecName "kube-api-access-dphl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:23.870074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.870045 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b69522a-eaa1-4839-aa5e-221caa4cef34-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:23.870074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.870070 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dphl9\" (UniqueName: \"kubernetes.io/projected/3b69522a-eaa1-4839-aa5e-221caa4cef34-kube-api-access-dphl9\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:23.923785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:23.923750 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3" path="/var/lib/kubelet/pods/88c9d245-cd3b-4501-adcc-9ec5b7d1b4a3/volumes" Nov 02 15:21:24.477715 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:24.477680 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" event={"ID":"3b69522a-eaa1-4839-aa5e-221caa4cef34","Type":"ContainerDied","Data":"8bb8e58208bcc9a4315f8b67d2fe9545a80d08c4649457b66ef9fe49f1729574"} Nov 02 15:21:24.477715 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:24.477709 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh" Nov 02 15:21:24.477893 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:24.477730 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bb8e58208bcc9a4315f8b67d2fe9545a80d08c4649457b66ef9fe49f1729574" Nov 02 15:21:27.304485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.304452 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp"] Nov 02 15:21:27.305326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.305305 5476 cpu_manager.go:414] "containerMap: removing container" podUID="7f5f7022-ae6a-4ecb-a28e-489226637f0e" containerName="revoke-keys" Nov 02 15:21:27.305432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.305326 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5f7022-ae6a-4ecb-a28e-489226637f0e" containerName="revoke-keys" Nov 02 15:21:27.305432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.305416 5476 cpu_manager.go:414] "containerMap: removing container" podUID="3b69522a-eaa1-4839-aa5e-221caa4cef34" containerName="revoke-keys" Nov 02 15:21:27.305432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.305425 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b69522a-eaa1-4839-aa5e-221caa4cef34" containerName="revoke-keys" Nov 02 15:21:27.305599 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.305585 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b69522a-eaa1-4839-aa5e-221caa4cef34" containerName="revoke-keys" Nov 02 15:21:27.305659 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.305605 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5f7022-ae6a-4ecb-a28e-489226637f0e" containerName="revoke-keys" Nov 02 15:21:27.311826 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.311785 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.335864 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.335827 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.336151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.336129 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.336333 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.336317 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.336456 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.336442 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.336619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.336605 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.336744 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.336730 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.336903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.336884 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.337193 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.337170 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slpsl\" (UniqueName: \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-kube-api-access-slpsl\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.337340 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.337322 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f36f5a9-b45a-4c86-986e-3a494915914b-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.438828 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.438790 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slpsl\" (UniqueName: \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-kube-api-access-slpsl\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.438828 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.438832 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f36f5a9-b45a-4c86-986e-3a494915914b-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.439122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.438901 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.439122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.438943 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.439122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.438978 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.439122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.439015 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.439122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.439094 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.439487 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.439369 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.439487 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.439444 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.439594 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.439503 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.439594 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.439539 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.439780 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.439754 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f36f5a9-b45a-4c86-986e-3a494915914b-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.439949 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.439875 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.440081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.439904 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.446274 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.446247 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.446627 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.446595 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.499684 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.499652 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.507493 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.507468 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slpsl\" (UniqueName: \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-kube-api-access-slpsl\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:27.525116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.522201 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp"] Nov 02 15:21:27.643964 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:27.643930 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:28.080707 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.080620 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp"] Nov 02 15:21:28.084380 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:28.084350 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f36f5a9_b45a_4c86_986e_3a494915914b.slice/crio-0ba38aa0a56460db235623d8d4a0719c86f2b11f0f45c602f4cf0638d3897eed WatchSource:0}: Error finding container 0ba38aa0a56460db235623d8d4a0719c86f2b11f0f45c602f4cf0638d3897eed: Status 404 returned error can't find the container with id 0ba38aa0a56460db235623d8d4a0719c86f2b11f0f45c602f4cf0638d3897eed Nov 02 15:21:28.086924 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.086890 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:28.087037 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.086958 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:28.087037 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.086987 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:28.518282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.518151 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" event={"ID":"0f36f5a9-b45a-4c86-986e-3a494915914b","Type":"ContainerStarted","Data":"8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4"} Nov 02 15:21:28.518770 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.518302 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" event={"ID":"0f36f5a9-b45a-4c86-986e-3a494915914b","Type":"ContainerStarted","Data":"0ba38aa0a56460db235623d8d4a0719c86f2b11f0f45c602f4cf0638d3897eed"} Nov 02 15:21:28.644724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.644695 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:28.826501 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.826409 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" podStartSLOduration=1.826394827 podStartE2EDuration="1.826394827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:28.694266885 +0000 UTC m=+1390.897144960" watchObservedRunningTime="2026-11-02 15:21:28.826394827 +0000 UTC m=+1391.029272886" Nov 02 15:21:28.828478 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.828455 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5"] Nov 02 15:21:28.833463 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.833445 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.854848 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.854821 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.854951 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.854858 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rprw\" (UniqueName: \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-kube-api-access-8rprw\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.855015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.854947 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.855061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.855014 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.855156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.855139 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.855192 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.855172 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.855261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.855249 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.855302 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.855272 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.855354 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.855343 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.923786 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.923753 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5"] Nov 02 15:21:28.956896 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.956860 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.957120 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.957074 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.957215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.957124 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rprw\" (UniqueName: \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-kube-api-access-8rprw\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.957215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.957191 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.957322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.957271 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.957418 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.957399 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.957473 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.957445 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.957533 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.957520 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.957588 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.957549 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.957870 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.957846 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.957924 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.957880 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.958510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.958446 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.958510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.958459 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.958660 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.958540 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.959875 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.959849 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:28.960798 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:28.960779 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:29.018184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.018155 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:29.018859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.018830 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rprw\" (UniqueName: \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-kube-api-access-8rprw\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:29.128217 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.128182 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb"] Nov 02 15:21:29.133899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.133875 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.148890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.148869 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:29.160663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.160626 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779804e1-968d-464a-83f3-648de3d24208-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.160792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.160677 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.160792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.160741 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.160792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.160769 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.160969 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.160838 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5nw\" (UniqueName: \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-kube-api-access-2p5nw\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.160969 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.160965 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779804e1-968d-464a-83f3-648de3d24208-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.161195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.161176 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.161258 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.161243 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.161311 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.161266 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.194034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.192342 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5"] Nov 02 15:21:29.207019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.206978 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb"] Nov 02 15:21:29.207187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.207100 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.262975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.262409 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.262975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.262495 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5nw\" (UniqueName: \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-kube-api-access-2p5nw\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.262975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.262564 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.262975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.262591 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.263315 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263060 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779804e1-968d-464a-83f3-648de3d24208-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.263315 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263120 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.263315 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263183 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.263315 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263221 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.263315 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263271 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.263315 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263314 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea558aff-5d34-4e17-9194-4cefdf5c266e-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.263634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263347 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8md4c\" (UniqueName: \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-kube-api-access-8md4c\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.263634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263382 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.263634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263410 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.263634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263437 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.263634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263504 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779804e1-968d-464a-83f3-648de3d24208-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.263634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263535 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.263634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263611 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.264010 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.263642 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.265638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.264145 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.265638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.264471 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.265638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.265057 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.265638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.265359 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.265638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.265423 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779804e1-968d-464a-83f3-648de3d24208-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.268122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.267847 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5"] Nov 02 15:21:29.270141 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.269860 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779804e1-968d-464a-83f3-648de3d24208-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.270141 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.270099 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.320713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.320675 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5nw\" (UniqueName: \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-kube-api-access-2p5nw\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.322191 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.322143 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.364839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.364799 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.365913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.364884 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.365913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.364953 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.365913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.364980 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea558aff-5d34-4e17-9194-4cefdf5c266e-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.365913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.365046 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8md4c\" (UniqueName: \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-kube-api-access-8md4c\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.365913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.365085 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.365913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.365219 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.365913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.365334 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.365913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.365457 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.365913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.365655 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.365913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.365788 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.365913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.365826 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.366334 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.365924 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.366334 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.365947 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea558aff-5d34-4e17-9194-4cefdf5c266e-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.368132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.368101 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.368132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.368118 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.424404 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.424333 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8md4c\" (UniqueName: \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-kube-api-access-8md4c\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.424542 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.424397 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.449754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.449722 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:29.526514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.526487 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:29.616250 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.616205 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl"] Nov 02 15:21:29.623680 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.622976 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.642469 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.642436 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5"] Nov 02 15:21:29.646081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.646049 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:21:29.646217 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.646101 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:21:29.652455 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:29.652396 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bf0ff2_c698_4896_a0bc_a32bab520c1a.slice/crio-9a19206e2eeb2ef513db7ae95a60638c03bcf6ab26148b38584c6de1705caab8 WatchSource:0}: Error finding container 9a19206e2eeb2ef513db7ae95a60638c03bcf6ab26148b38584c6de1705caab8: Status 404 returned error can't find the container with id 9a19206e2eeb2ef513db7ae95a60638c03bcf6ab26148b38584c6de1705caab8 Nov 02 15:21:29.655216 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.655179 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.655327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.655264 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.655327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.655319 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.714086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.712327 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.714086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.712431 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.714086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.712587 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.714086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.712667 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpszx\" (UniqueName: \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-kube-api-access-dpszx\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.714086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.712717 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.714086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.712780 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.714086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.712833 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.714086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.712884 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.714086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.712924 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.721166 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.721136 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl"] Nov 02 15:21:29.814180 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.814149 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.814304 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.814252 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.814408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.814390 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.814483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.814471 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpszx\" (UniqueName: \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-kube-api-access-dpszx\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.814605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.814522 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.814605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.814548 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.815323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.814752 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.815323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.814813 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.815323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.814867 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.815323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.814904 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.815323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.814933 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.815323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.815201 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.815323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.815275 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.815741 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.815716 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.817543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.817513 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.818243 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.818205 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.902034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.877180 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb"] Nov 02 15:21:29.904778 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.904713 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpszx\" (UniqueName: \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-kube-api-access-dpszx\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.905411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.905345 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:29.906089 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:29.906058 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod779804e1_968d_464a_83f3_648de3d24208.slice/crio-838b29c8d1e6843dd4048dceb9c9235c0e75404c08cdb58f8cfad133aa327382 WatchSource:0}: Error finding container 838b29c8d1e6843dd4048dceb9c9235c0e75404c08cdb58f8cfad133aa327382: Status 404 returned error can't find the container with id 838b29c8d1e6843dd4048dceb9c9235c0e75404c08cdb58f8cfad133aa327382 Nov 02 15:21:29.908559 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.908529 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.908637 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.908594 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.908677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.908638 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.944470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:29.944447 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:30.021667 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.021632 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5"] Nov 02 15:21:30.027825 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:30.027783 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea558aff_5d34_4e17_9194_4cefdf5c266e.slice/crio-d917ffaeef14ffb3abd21fac0202dd8de21c61e85a5aedd25b3efcbab17b0be8 WatchSource:0}: Error finding container d917ffaeef14ffb3abd21fac0202dd8de21c61e85a5aedd25b3efcbab17b0be8: Status 404 returned error can't find the container with id d917ffaeef14ffb3abd21fac0202dd8de21c61e85a5aedd25b3efcbab17b0be8 Nov 02 15:21:30.030408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.030366 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.030537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.030443 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.030537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.030474 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.069207 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.069146 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr"] Nov 02 15:21:30.120696 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.120658 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc31003-pf6lr"] Nov 02 15:21:30.411976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.402694 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl"] Nov 02 15:21:30.429036 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.426826 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.429036 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.426914 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.429036 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.426956 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.659104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.658838 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:21:30.659104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.658891 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:21:30.659618 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.659526 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" event={"ID":"779804e1-968d-464a-83f3-648de3d24208","Type":"ContainerStarted","Data":"decdd28a346334c8dcd0f1f19ff3b2aee165176bec9f9a575bdffae832b13295"} Nov 02 15:21:30.659618 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.659580 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" event={"ID":"779804e1-968d-464a-83f3-648de3d24208","Type":"ContainerStarted","Data":"838b29c8d1e6843dd4048dceb9c9235c0e75404c08cdb58f8cfad133aa327382"} Nov 02 15:21:30.730020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.704170 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" event={"ID":"a6c27ff3-82e0-4411-ab70-c2189b9cf795","Type":"ContainerStarted","Data":"e6f7553f19b740dc0e79305cb67ce83d3df9a0b6a9412ec9a22ff129aa52899c"} Nov 02 15:21:30.740090 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.736122 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" event={"ID":"b7bf0ff2-c698-4896-a0bc-a32bab520c1a","Type":"ContainerStarted","Data":"da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172"} Nov 02 15:21:30.740090 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.736185 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" event={"ID":"b7bf0ff2-c698-4896-a0bc-a32bab520c1a","Type":"ContainerStarted","Data":"9a19206e2eeb2ef513db7ae95a60638c03bcf6ab26148b38584c6de1705caab8"} Nov 02 15:21:30.753019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.749684 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" event={"ID":"ea558aff-5d34-4e17-9194-4cefdf5c266e","Type":"ContainerStarted","Data":"380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b"} Nov 02 15:21:30.753019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.749742 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" event={"ID":"ea558aff-5d34-4e17-9194-4cefdf5c266e","Type":"ContainerStarted","Data":"d917ffaeef14ffb3abd21fac0202dd8de21c61e85a5aedd25b3efcbab17b0be8"} Nov 02 15:21:30.832226 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:30.832169 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podStartSLOduration=1.8321493979999999 podStartE2EDuration="1.832149398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:30.830141157 +0000 UTC m=+1393.033019218" watchObservedRunningTime="2026-11-02 15:21:30.832149398 +0000 UTC m=+1393.035027460" Nov 02 15:21:31.045071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.031909 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podStartSLOduration=3.031887544 podStartE2EDuration="3.031887544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:30.960451571 +0000 UTC m=+1393.163329632" watchObservedRunningTime="2026-11-02 15:21:31.031887544 +0000 UTC m=+1393.234765605" Nov 02 15:21:31.092348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.092284 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podStartSLOduration=2.092264107 podStartE2EDuration="2.092264107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:31.09127646 +0000 UTC m=+1393.294154520" watchObservedRunningTime="2026-11-02 15:21:31.092264107 +0000 UTC m=+1393.295142169" Nov 02 15:21:31.151638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.151141 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:31.151638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.151292 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:21:31.151638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.151339 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:21:31.456020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.452862 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:21:31.456020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.452938 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:21:31.456020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.453304 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:31.520134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.520084 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:31.532186 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.532154 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:31.532524 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.532502 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 15:21:31.532697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.532672 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 15:21:31.548142 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.548100 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:31.646690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.646655 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:21:31.646936 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.646912 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:21:31.888952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.872978 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" event={"ID":"a6c27ff3-82e0-4411-ab70-c2189b9cf795","Type":"ContainerStarted","Data":"dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2"} Nov 02 15:21:31.969021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.968090 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:21:31.976025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:31.970276 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:32.000714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.000674 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.006531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.004883 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podStartSLOduration=3.004863845 podStartE2EDuration="3.004863845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:32.002847926 +0000 UTC m=+1394.205725987" watchObservedRunningTime="2026-11-02 15:21:32.004863845 +0000 UTC m=+1394.207741905" Nov 02 15:21:32.045062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.006243 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9acbf997-dc21-402e-8567-639b5a819aa7" path="/var/lib/kubelet/pods/9acbf997-dc21-402e-8567-639b5a819aa7/volumes" Nov 02 15:21:32.045062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.019626 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:32.045062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.006516 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 15:21:32.153078 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.152441 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.153078 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.152508 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:21:32.306018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.288242 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:32.452073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.451372 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:21:32.460019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.457952 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.460019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.458043 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:21:32.494163 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.494111 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:32.494738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.494111 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:32.499313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.499273 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:21:32.521543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.521501 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:32.606063 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.521757 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:32.613029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.606841 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:21:32.647018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.521825 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:32.647018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.544495 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.647018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.618890 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 15:21:32.647018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.557813 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:32.647018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.619365 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:21:32.653016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.649151 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.653298 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.653269 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:21:33.061121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.984089 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:21:33.061121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:32.984595 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:21:33.121310 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.114147 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.148020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.140171 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:21:33.185018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.152793 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.185018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.172089 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 15:21:33.185018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.174930 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.185018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.174975 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:21:33.599123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.291420 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.599123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.336392 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:21:33.599123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.579066 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:21:33.599123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.579399 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:21:33.786031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.780561 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 02 15:21:33.786031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.781780 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:21:33.817038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.786717 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:21:33.817038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.786780 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:21:33.817038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.787173 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.817038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.787660 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:21:33.817038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.789511 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 15:21:33.817038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.789695 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded" Nov 02 15:21:33.817038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.789780 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:21:33.817038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.804882 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.817038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.805123 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:21:33.817038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.805283 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:21:33.817038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.805470 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:21:33.839071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.828361 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.839071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.828417 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:21:33.839071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.828496 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.839071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.828519 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 15:21:33.875274 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.875237 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.875548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.875519 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:21:33.997180 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:33.997136 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.035021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.010106 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 15:21:34.155082 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.130041 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.155082 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.142307 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.196021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.156647 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.196021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.156725 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:21:34.403020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.386530 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.485724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.485628 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.486038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.485980 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:21:34.527035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.526980 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.527332 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.527291 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 15:21:34.589016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.587719 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.697022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.653773 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.697323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.697283 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:21:34.749575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.746676 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4"] Nov 02 15:21:34.760518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.760486 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dcd3cb-7drm4"] Nov 02 15:21:34.807019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.786242 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.808880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.808839 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.837018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.811790 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 15:21:34.837018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.812367 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.867019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.812400 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.942204 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:34.812439 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.041117 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.027172 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.099926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.099891 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.118600 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.118561 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 15:21:35.211019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.203082 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:21:35.211664 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.211636 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.212725 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.212689 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.212916 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.212891 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:21:35.539021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.501881 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.539329 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.510143 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.607583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.510189 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.607978 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.540698 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.608193 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.608156 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 15:21:35.609507 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.609471 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:21:35.613183 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.613156 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.685018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.683199 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.685018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.683287 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:21:35.710020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.703386 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded" Nov 02 15:21:35.858016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.831621 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.862017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.841139 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.904727 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.841196 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.905210 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.867952 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.905822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.898341 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.906366 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.904294 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.910294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.905542 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" podUID="6abac5be-31c8-4f53-a556-cf43374c83a9" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.085275 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:35.990199 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.085559 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.085518 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 15:21:36.114107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.101700 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.114107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.105939 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.171214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.171164 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.171517 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.171471 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:21:36.242022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.216679 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 02 15:21:36.295107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.236409 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:21:36.329625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.268616 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0361344-89da-461b-abed-07c1b63d997e" path="/var/lib/kubelet/pods/c0361344-89da-461b-abed-07c1b63d997e/volumes" Nov 02 15:21:36.363019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.293780 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.363019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.335791 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.506586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.506494 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.516268 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.516228 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:21:36.553015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.539157 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.604810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.604777 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.624064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.619973 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 15:21:36.624290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.618762 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.625737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.618708 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:21:36.647359 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.647326 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.647609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.647580 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 02 15:21:36.870703 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.870349 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.912466 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.912405 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.926544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.913102 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.976512 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.913289 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.977455 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.915704 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.985150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.915906 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.985912 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.958332 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.986138 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.986092 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 15:21:36.998229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:36.998176 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Nov 02 15:21:37.105019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.102903 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 02 15:21:37.160705 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.160601 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:21:37.161028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.160981 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:21:37.375117 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.361705 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:21:37.375117 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.362121 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:21:37.566076 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.565805 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 15:21:37.566076 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.565868 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 15:21:37.580861 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.578588 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:21:37.580861 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.578652 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:21:37.635027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.624612 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.661569 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.629642 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.000017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.971394 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:21:38.000017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.973024 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.000017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:37.973363 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.060743 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.057321 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:38.060743 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.057729 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:21:38.060743 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.058028 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 15:21:38.079230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.077831 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.079230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.077884 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 15:21:38.107695 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.106931 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.171461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.171421 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.171707 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.171679 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:21:38.537189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.498963 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:38.643199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.511470 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:21:38.643199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.642986 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:38.643199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.511504 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:21:38.643587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.513107 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.643587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.643355 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:21:38.643587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.610319 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.643587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.643414 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 15:21:38.643587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.643472 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.722561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.722524 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:21:38.861080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.790726 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Nov 02 15:21:38.991074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.944877 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:21:38.991074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.944976 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.991074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.967398 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 15:21:38.991074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.978469 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:21:38.991074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.978806 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:21:38.991074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:38.978408 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:39.058264 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.058227 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs"] Nov 02 15:21:39.231797 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.067977 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:39.231797 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.068698 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:39.231797 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.075305 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:21:39.231797 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.140513 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:39.254012 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.233818 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 15:21:39.254012 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.233866 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 15:21:39.341015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.333722 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:21:39.342724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.341608 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:21:39.387052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.386526 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d5e3966-9bzbs"] Nov 02 15:21:39.467989 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.439049 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:21:39.490493 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.490413 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:21:39.560837 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.560798 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:21:39.561479 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.560861 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:21:39.566703 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.566665 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 15:21:39.566891 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.566865 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 15:21:39.576847 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.576825 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:21:39.623407 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.623363 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:21:39.649653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.648975 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:39.660337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.660115 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:21:39.768364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.767337 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:21:39.847906 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.846727 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:21:39.899257 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.899229 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:21:39.956252 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.956217 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 15:21:39.956559 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:39.956513 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 15:21:40.035365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.035271 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee66e224-2fad-4719-bd75-51cd511cb603" path="/var/lib/kubelet/pods/ee66e224-2fad-4719-bd75-51cd511cb603/volumes" Nov 02 15:21:40.042088 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.042058 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:21:40.451063 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.451028 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:21:40.451320 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.451292 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 15:21:40.452941 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.452914 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz"] Nov 02 15:21:40.458728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.458703 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:40.501941 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.501913 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:40.503704 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.503686 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:40.504774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.504756 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:21:40.510731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.510714 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:21:40.513531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.513509 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:21:40.517854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.517825 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz"] Nov 02 15:21:40.552481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.546088 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 15:21:40.552481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.546146 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 15:21:40.622021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.621911 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbq9n\" (UniqueName: \"kubernetes.io/projected/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-kube-api-access-jbq9n\") pod \"maas-api-revoke-sub-7ee0de28-d8mmz\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:40.622503 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.622108 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee0de28-d8mmz\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:40.622503 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.622267 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee0de28-d8mmz\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:40.725068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.724339 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbq9n\" (UniqueName: \"kubernetes.io/projected/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-kube-api-access-jbq9n\") pod \"maas-api-revoke-sub-7ee0de28-d8mmz\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:40.725068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.724480 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee0de28-d8mmz\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:40.725068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.724592 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee0de28-d8mmz\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:40.727034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.725520 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee0de28-d8mmz\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:40.735770 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.735743 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee0de28-d8mmz\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:40.839045 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.831555 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbq9n\" (UniqueName: \"kubernetes.io/projected/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-kube-api-access-jbq9n\") pod \"maas-api-revoke-sub-7ee0de28-d8mmz\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:40.878761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.878732 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:40.952766 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.952642 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 02 15:21:40.952766 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:40.952698 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 02 15:21:41.393948 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:41.393793 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz"] Nov 02 15:21:41.409342 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:41.409311 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29c38e8_501b_4c00_8a8a_01f8e92fcc67.slice/crio-53679f68a9477dd1dcfb3e9dcd376bdedda277719c1e1ed04016f816fdd3b64c WatchSource:0}: Error finding container 53679f68a9477dd1dcfb3e9dcd376bdedda277719c1e1ed04016f816fdd3b64c: Status 404 returned error can't find the container with id 53679f68a9477dd1dcfb3e9dcd376bdedda277719c1e1ed04016f816fdd3b64c Nov 02 15:21:41.454574 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:41.454550 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:41.455118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:41.455101 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:41.455979 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:41.455952 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:21:41.532349 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:41.532322 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:41.532653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:41.532636 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:41.533377 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:41.533356 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:21:41.955888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:41.955854 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:41.957781 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:41.957597 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:41.958267 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:41.958245 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:21:42.260010 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:42.259960 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" event={"ID":"b29c38e8-501b-4c00-8a8a-01f8e92fcc67","Type":"ContainerStarted","Data":"2b76b5913fb5a2a6b540394dd6504637c15a7c9fae5d9e5edec5323db034892d"} Nov 02 15:21:42.260200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:42.260055 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" event={"ID":"b29c38e8-501b-4c00-8a8a-01f8e92fcc67","Type":"ContainerStarted","Data":"53679f68a9477dd1dcfb3e9dcd376bdedda277719c1e1ed04016f816fdd3b64c"} Nov 02 15:21:42.372908 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:42.371397 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" podStartSLOduration=2.371376566 podStartE2EDuration="2.371376566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:42.368431872 +0000 UTC m=+1404.571309934" watchObservedRunningTime="2026-11-02 15:21:42.371376566 +0000 UTC m=+1404.574254627" Nov 02 15:21:43.051362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:43.051325 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj"] Nov 02 15:21:43.056216 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:43.056187 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-167a5990-nb5qj"] Nov 02 15:21:43.270616 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:43.270578 5476 generic.go:356] "Generic (PLEG): container finished" podID="b29c38e8-501b-4c00-8a8a-01f8e92fcc67" containerID="2b76b5913fb5a2a6b540394dd6504637c15a7c9fae5d9e5edec5323db034892d" exitCode=0 Nov 02 15:21:43.270800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:43.270662 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" event={"ID":"b29c38e8-501b-4c00-8a8a-01f8e92fcc67","Type":"ContainerDied","Data":"2b76b5913fb5a2a6b540394dd6504637c15a7c9fae5d9e5edec5323db034892d"} Nov 02 15:21:43.926589 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:43.926542 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe12e16c-99b0-443b-a5c3-10b4919cb675" path="/var/lib/kubelet/pods/fe12e16c-99b0-443b-a5c3-10b4919cb675/volumes" Nov 02 15:21:44.055840 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.055806 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2"] Nov 02 15:21:44.062380 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.062355 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe90abe-pt6c2"] Nov 02 15:21:44.507175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.507145 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45"] Nov 02 15:21:44.520022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.516246 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:44.705316 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.691524 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45"] Nov 02 15:21:44.752129 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.730521 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d70a7ef0-ttt45\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:44.752129 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.730618 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d70a7ef0-ttt45\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:44.752129 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.730726 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxxbr\" (UniqueName: \"kubernetes.io/projected/a538d791-3c71-47d0-82aa-f11762f4470f-kube-api-access-sxxbr\") pod \"maas-api-revoke-sub-d70a7ef0-ttt45\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:44.767838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.767809 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:44.831616 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.831579 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-kube-api-access-jbq9n\" (UniqueName: \"kubernetes.io/projected/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-kube-api-access-jbq9n\") pod \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " Nov 02 15:21:44.845097 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.845068 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-openshift-ca\") pod \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " Nov 02 15:21:44.845809 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.845788 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-xks-ca\") pod \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\" (UID: \"b29c38e8-501b-4c00-8a8a-01f8e92fcc67\") " Nov 02 15:21:44.846702 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.846673 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d70a7ef0-ttt45\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:44.857060 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.857033 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d70a7ef0-ttt45\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:44.857312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.857298 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxxbr\" (UniqueName: \"kubernetes.io/projected/a538d791-3c71-47d0-82aa-f11762f4470f-kube-api-access-sxxbr\") pod \"maas-api-revoke-sub-d70a7ef0-ttt45\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:44.857665 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.845610 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-openshift-ca" pod "b29c38e8-501b-4c00-8a8a-01f8e92fcc67" (UID: "b29c38e8-501b-4c00-8a8a-01f8e92fcc67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:44.880463 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.858472 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d70a7ef0-ttt45\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:44.880463 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.873446 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d70a7ef0-ttt45\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:44.899074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.880968 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-kube-api-access-jbq9n" pod "b29c38e8-501b-4c00-8a8a-01f8e92fcc67" (UID: "b29c38e8-501b-4c00-8a8a-01f8e92fcc67"). InnerVolumeSpecName "kube-api-access-jbq9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:44.926016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.922347 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-xks-ca" pod "b29c38e8-501b-4c00-8a8a-01f8e92fcc67" (UID: "b29c38e8-501b-4c00-8a8a-01f8e92fcc67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:44.982339 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.982218 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:44.982339 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.982261 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbq9n\" (UniqueName: \"kubernetes.io/projected/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-kube-api-access-jbq9n\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:44.982339 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:44.982279 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29c38e8-501b-4c00-8a8a-01f8e92fcc67-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:45.184123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.176721 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxxbr\" (UniqueName: \"kubernetes.io/projected/a538d791-3c71-47d0-82aa-f11762f4470f-kube-api-access-sxxbr\") pod \"maas-api-revoke-sub-d70a7ef0-ttt45\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:45.185155 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.184866 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:45.238824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.238790 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw"] Nov 02 15:21:45.240135 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.240109 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b29c38e8-501b-4c00-8a8a-01f8e92fcc67" containerName="revoke-keys" Nov 02 15:21:45.240285 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.240270 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29c38e8-501b-4c00-8a8a-01f8e92fcc67" containerName="revoke-keys" Nov 02 15:21:45.240709 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.240693 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29c38e8-501b-4c00-8a8a-01f8e92fcc67" containerName="revoke-keys" Nov 02 15:21:45.260280 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.260251 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:45.358076 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.348940 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw"] Nov 02 15:21:45.434028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.430659 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd3ebca-lhmhw\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:45.434028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.430733 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpw9w\" (UniqueName: \"kubernetes.io/projected/de02b6ea-99c6-4047-827a-909cc2d60367-kube-api-access-qpw9w\") pod \"maas-api-revoke-sub-efd3ebca-lhmhw\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:45.434028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.430839 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd3ebca-lhmhw\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:45.631022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.616900 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd3ebca-lhmhw\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:45.713042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.616981 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpw9w\" (UniqueName: \"kubernetes.io/projected/de02b6ea-99c6-4047-827a-909cc2d60367-kube-api-access-qpw9w\") pod \"maas-api-revoke-sub-efd3ebca-lhmhw\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:45.713431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.713406 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd3ebca-lhmhw\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:45.739331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.714307 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd3ebca-lhmhw\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:45.938021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.880577 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpw9w\" (UniqueName: \"kubernetes.io/projected/de02b6ea-99c6-4047-827a-909cc2d60367-kube-api-access-qpw9w\") pod \"maas-api-revoke-sub-efd3ebca-lhmhw\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:46.160104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.941797 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" event={"ID":"b29c38e8-501b-4c00-8a8a-01f8e92fcc67","Type":"ContainerDied","Data":"53679f68a9477dd1dcfb3e9dcd376bdedda277719c1e1ed04016f816fdd3b64c"} Nov 02 15:21:46.160440 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.160417 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53679f68a9477dd1dcfb3e9dcd376bdedda277719c1e1ed04016f816fdd3b64c" Nov 02 15:21:46.160624 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.846507 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd3ebca-lhmhw\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:46.160748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:45.941884 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz" Nov 02 15:21:46.200019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.182306 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.200019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.193894 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.295017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.284500 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:46.322668 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.310107 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.353378 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.351518 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b1edaf-e52e-42be-9411-ff8fc4a7e8f1" path="/var/lib/kubelet/pods/21b1edaf-e52e-42be-9411-ff8fc4a7e8f1/volumes" Nov 02 15:21:46.637150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.637102 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.681143 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.681045 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": context deadline exceeded" Nov 02 15:21:46.754174 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.754123 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.811421 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.810106 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.837730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.812174 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.897509 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.897471 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:21:47.019364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.019260 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 15:21:47.023549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.023516 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Nov 02 15:21:47.054448 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:46.921217 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.054448 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.026434 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.054448 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.053797 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.213261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.209051 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:21:47.213261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.209391 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:21:47.215801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.214385 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:21:47.653275 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.653141 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.698967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.665209 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:21:47.781502 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.767442 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.781502 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.767746 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:21:48.031044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.854091 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:21:48.031044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.874092 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:21:48.031044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.923155 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:21:48.031044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:47.953482 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:21:48.031044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:48.019893 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:21:48.032823 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:48.031490 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.032952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:48.032841 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.042577 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:48.042174 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.066557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:48.065070 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.239147 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:48.239099 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.276020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:48.268451 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:21:48.752208 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:48.752160 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.947019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:48.914320 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Nov 02 15:21:49.017310 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.017270 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 15:21:49.118782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.115063 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:21:49.118782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.115456 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.159123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.156373 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.181350 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.181285 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.182725 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.182694 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:49.359706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.351135 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 02 15:21:49.415516 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.361974 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.682092 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.672344 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 15:21:49.813042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.793524 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:21:49.813042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.811251 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:49.813042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.811311 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.901020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.895080 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:49.901020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.895162 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.049084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.922365 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.049084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:49.957503 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:21:50.049084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:50.036553 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.265022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:50.208155 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.361081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:50.278186 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:21:50.361081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:50.278237 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 15:21:50.361081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:50.278289 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.361081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:50.278417 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.361081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:50.358118 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.361081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:50.358239 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.387031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:50.383096 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:21:50.419168 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:50.408126 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" podUID="6abac5be-31c8-4f53-a556-cf43374c83a9" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:51.084082 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.059932 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:21:51.089057 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.084359 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:51.245349 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.244181 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:51.245349 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.244249 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:51.346798 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.267896 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:21:51.380026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.359456 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:21:51.380026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.359876 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:51.380026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.365675 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:21:51.380026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.365922 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:21:51.380026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.366169 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 15:21:51.392038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.391966 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:51.393470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.393421 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 15:21:51.424318 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.416758 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:51.424318 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.416821 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:51.458015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:51.457689 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:21:51.879020 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:21:51.878379 5476 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.924s" Nov 02 15:21:52.067512 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.067410 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:52.145019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.122362 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:21:52.269153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.269099 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:52.284206 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.270944 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:52.284206 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.271022 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:52.523136 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:21:52.523088 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:21:52.573360 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.573248 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:52.574801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.574775 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:21:52.575935 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.575911 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:52.620538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.620328 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:52.620538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.620396 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" podUID="a1549618-289f-426e-997d-277d2705d0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:52.642646 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.639220 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" event={"ID":"a538d791-3c71-47d0-82aa-f11762f4470f","Type":"ContainerStarted","Data":"9966198a4724d681a1810f152c5b1bddf592eb685c5009f720e16d9c9dc9d281"} Nov 02 15:21:52.654062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.652327 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45"] Nov 02 15:21:52.654062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.652377 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw"] Nov 02 15:21:52.654062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.652448 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:21:52.654062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.652495 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:21:52.654062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.652527 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:21:52.654062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.652561 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:21:52.654062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.652598 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:21:52.654062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.652628 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:21:52.654062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.652654 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:21:52.661018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.655523 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 15:21:52.661018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.657017 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 15:21:52.661018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.657966 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 15:21:52.661018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.659139 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:21:52.662042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.661986 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:52.713012 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.710358 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:21:52.743374 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.735831 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:21:52.751164 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.742808 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:21:52.768833 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.768805 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:21:52.769090 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.769074 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:21:52.802321 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.802296 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:21:52.805226 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.805205 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:21:52.810607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.808582 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:21:52.819301 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.819270 5476 generic.go:356] "Generic (PLEG): container finished" podID="a538d791-3c71-47d0-82aa-f11762f4470f" containerID="27278495d214903f8e313956bd32482b9deb4802ca44b3a65bd734a2f598e50b" exitCode=0 Nov 02 15:21:52.819408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.819359 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" event={"ID":"a538d791-3c71-47d0-82aa-f11762f4470f","Type":"ContainerDied","Data":"27278495d214903f8e313956bd32482b9deb4802ca44b3a65bd734a2f598e50b"} Nov 02 15:21:52.821112 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:52.821091 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" event={"ID":"de02b6ea-99c6-4047-827a-909cc2d60367","Type":"ContainerStarted","Data":"2b57833278483262c2ca770d43be472d4dc7f1eafa93fe0a57ddce0aef44e6ce"} Nov 02 15:21:53.832676 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:53.832581 5476 generic.go:356] "Generic (PLEG): container finished" podID="de02b6ea-99c6-4047-827a-909cc2d60367" containerID="e90ca589223ce5c7036b7d9039ada9746e8d18bd3fd6a5bd80219501682268fb" exitCode=0 Nov 02 15:21:53.833215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:53.832727 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" event={"ID":"de02b6ea-99c6-4047-827a-909cc2d60367","Type":"ContainerDied","Data":"e90ca589223ce5c7036b7d9039ada9746e8d18bd3fd6a5bd80219501682268fb"} Nov 02 15:21:53.892733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:53.892693 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b"] Nov 02 15:21:53.892917 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:53.892896 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerName="istio-proxy" containerID="cri-o://47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99" gracePeriod=30 Nov 02 15:21:54.031094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.031062 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:54.172975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.172931 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-openshift-ca\") pod \"a538d791-3c71-47d0-82aa-f11762f4470f\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " Nov 02 15:21:54.173203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.173103 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a538d791-3c71-47d0-82aa-f11762f4470f-kube-api-access-sxxbr\" (UniqueName: \"kubernetes.io/projected/a538d791-3c71-47d0-82aa-f11762f4470f-kube-api-access-sxxbr\") pod \"a538d791-3c71-47d0-82aa-f11762f4470f\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " Nov 02 15:21:54.173203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.173156 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-xks-ca\") pod \"a538d791-3c71-47d0-82aa-f11762f4470f\" (UID: \"a538d791-3c71-47d0-82aa-f11762f4470f\") " Nov 02 15:21:54.174774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.174742 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-openshift-ca" pod "a538d791-3c71-47d0-82aa-f11762f4470f" (UID: "a538d791-3c71-47d0-82aa-f11762f4470f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:54.176719 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.176688 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-xks-ca" pod "a538d791-3c71-47d0-82aa-f11762f4470f" (UID: "a538d791-3c71-47d0-82aa-f11762f4470f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:54.176859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.176833 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a538d791-3c71-47d0-82aa-f11762f4470f-kube-api-access-sxxbr" pod "a538d791-3c71-47d0-82aa-f11762f4470f" (UID: "a538d791-3c71-47d0-82aa-f11762f4470f"). InnerVolumeSpecName "kube-api-access-sxxbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:54.181459 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.181388 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds"] Nov 02 15:21:54.182729 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.182706 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a538d791-3c71-47d0-82aa-f11762f4470f" containerName="revoke-keys" Nov 02 15:21:54.182880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.182732 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a538d791-3c71-47d0-82aa-f11762f4470f" containerName="revoke-keys" Nov 02 15:21:54.183077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.183058 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a538d791-3c71-47d0-82aa-f11762f4470f" containerName="revoke-keys" Nov 02 15:21:54.187492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.187469 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:54.248839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.248808 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds"] Nov 02 15:21:54.275260 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.275218 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a3df04-drjds\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:54.275557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.275321 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4xnd\" (UniqueName: \"kubernetes.io/projected/dc86d00f-2713-4d37-993d-63e87d23f495-kube-api-access-l4xnd\") pod \"maas-api-revoke-sub-83a3df04-drjds\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:54.275557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.275550 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a3df04-drjds\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:54.275958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.275629 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:54.275958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.275646 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxxbr\" (UniqueName: \"kubernetes.io/projected/a538d791-3c71-47d0-82aa-f11762f4470f-kube-api-access-sxxbr\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:54.275958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.275661 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a538d791-3c71-47d0-82aa-f11762f4470f-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:54.377070 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.377031 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a3df04-drjds\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:54.377252 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.377131 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4xnd\" (UniqueName: \"kubernetes.io/projected/dc86d00f-2713-4d37-993d-63e87d23f495-kube-api-access-l4xnd\") pod \"maas-api-revoke-sub-83a3df04-drjds\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:54.377325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.377309 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a3df04-drjds\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:54.377782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.377763 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83a3df04-drjds\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:54.379678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.379660 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83a3df04-drjds\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:54.440583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.440508 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4xnd\" (UniqueName: \"kubernetes.io/projected/dc86d00f-2713-4d37-993d-63e87d23f495-kube-api-access-l4xnd\") pod \"maas-api-revoke-sub-83a3df04-drjds\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:54.504670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.504615 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" podStartSLOduration=9.504601737 podStartE2EDuration="9.504601737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:54.50019286 +0000 UTC m=+1416.703070919" watchObservedRunningTime="2026-11-02 15:21:54.504601737 +0000 UTC m=+1416.707479796" Nov 02 15:21:54.506611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.506592 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:54.843714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.843608 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" Nov 02 15:21:54.843714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.843617 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45" event={"ID":"a538d791-3c71-47d0-82aa-f11762f4470f","Type":"ContainerDied","Data":"9966198a4724d681a1810f152c5b1bddf592eb685c5009f720e16d9c9dc9d281"} Nov 02 15:21:54.843714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.843701 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9966198a4724d681a1810f152c5b1bddf592eb685c5009f720e16d9c9dc9d281" Nov 02 15:21:54.896244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:54.896158 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds"] Nov 02 15:21:54.901873 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:54.901242 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc86d00f_2713_4d37_993d_63e87d23f495.slice/crio-c85bf0ff3703bbfe01e8ed370ae90ef3e9093ae643aadf426ce9568f97e4a25e WatchSource:0}: Error finding container c85bf0ff3703bbfe01e8ed370ae90ef3e9093ae643aadf426ce9568f97e4a25e: Status 404 returned error can't find the container with id c85bf0ff3703bbfe01e8ed370ae90ef3e9093ae643aadf426ce9568f97e4a25e Nov 02 15:21:55.082915 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.082888 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:55.195134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.195100 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de02b6ea-99c6-4047-827a-909cc2d60367-kube-api-access-qpw9w\" (UniqueName: \"kubernetes.io/projected/de02b6ea-99c6-4047-827a-909cc2d60367-kube-api-access-qpw9w\") pod \"de02b6ea-99c6-4047-827a-909cc2d60367\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " Nov 02 15:21:55.195322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.195208 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-openshift-ca\") pod \"de02b6ea-99c6-4047-827a-909cc2d60367\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " Nov 02 15:21:55.195322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.195236 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-xks-ca\") pod \"de02b6ea-99c6-4047-827a-909cc2d60367\" (UID: \"de02b6ea-99c6-4047-827a-909cc2d60367\") " Nov 02 15:21:55.195918 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.195878 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-openshift-ca" pod "de02b6ea-99c6-4047-827a-909cc2d60367" (UID: "de02b6ea-99c6-4047-827a-909cc2d60367"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:55.199839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.199808 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de02b6ea-99c6-4047-827a-909cc2d60367-kube-api-access-qpw9w" pod "de02b6ea-99c6-4047-827a-909cc2d60367" (UID: "de02b6ea-99c6-4047-827a-909cc2d60367"). InnerVolumeSpecName "kube-api-access-qpw9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:55.199943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.199857 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-xks-ca" pod "de02b6ea-99c6-4047-827a-909cc2d60367" (UID: "de02b6ea-99c6-4047-827a-909cc2d60367"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:55.297102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.297076 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpw9w\" (UniqueName: \"kubernetes.io/projected/de02b6ea-99c6-4047-827a-909cc2d60367-kube-api-access-qpw9w\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:55.297102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.297102 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:55.297255 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.297112 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de02b6ea-99c6-4047-827a-909cc2d60367-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:55.855969 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.855945 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" Nov 02 15:21:55.856449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.856021 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw" event={"ID":"de02b6ea-99c6-4047-827a-909cc2d60367","Type":"ContainerDied","Data":"2b57833278483262c2ca770d43be472d4dc7f1eafa93fe0a57ddce0aef44e6ce"} Nov 02 15:21:55.856449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.856083 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b57833278483262c2ca770d43be472d4dc7f1eafa93fe0a57ddce0aef44e6ce" Nov 02 15:21:55.857950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.857925 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" event={"ID":"dc86d00f-2713-4d37-993d-63e87d23f495","Type":"ContainerStarted","Data":"b4aa070bf6e2411cb2fc31deafa2850d01fe54577977658f5cebb9549663b7d4"} Nov 02 15:21:55.858072 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:55.857975 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" event={"ID":"dc86d00f-2713-4d37-993d-63e87d23f495","Type":"ContainerStarted","Data":"c85bf0ff3703bbfe01e8ed370ae90ef3e9093ae643aadf426ce9568f97e4a25e"} Nov 02 15:21:56.868161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:56.868125 5476 generic.go:356] "Generic (PLEG): container finished" podID="dc86d00f-2713-4d37-993d-63e87d23f495" containerID="b4aa070bf6e2411cb2fc31deafa2850d01fe54577977658f5cebb9549663b7d4" exitCode=0 Nov 02 15:21:56.868629 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:56.868181 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" event={"ID":"dc86d00f-2713-4d37-993d-63e87d23f495","Type":"ContainerDied","Data":"b4aa070bf6e2411cb2fc31deafa2850d01fe54577977658f5cebb9549663b7d4"} Nov 02 15:21:57.029428 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.029402 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:57.124435 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.124347 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc86d00f-2713-4d37-993d-63e87d23f495-kube-api-access-l4xnd\" (UniqueName: \"kubernetes.io/projected/dc86d00f-2713-4d37-993d-63e87d23f495-kube-api-access-l4xnd\") pod \"dc86d00f-2713-4d37-993d-63e87d23f495\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " Nov 02 15:21:57.124435 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.124399 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-xks-ca\") pod \"dc86d00f-2713-4d37-993d-63e87d23f495\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " Nov 02 15:21:57.124671 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.124441 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-openshift-ca\") pod \"dc86d00f-2713-4d37-993d-63e87d23f495\" (UID: \"dc86d00f-2713-4d37-993d-63e87d23f495\") " Nov 02 15:21:57.124960 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.124928 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-openshift-ca" pod "dc86d00f-2713-4d37-993d-63e87d23f495" (UID: "dc86d00f-2713-4d37-993d-63e87d23f495"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:57.125499 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.125473 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:57.126617 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.126594 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-xks-ca" pod "dc86d00f-2713-4d37-993d-63e87d23f495" (UID: "dc86d00f-2713-4d37-993d-63e87d23f495"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:57.126767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.126742 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc86d00f-2713-4d37-993d-63e87d23f495-kube-api-access-l4xnd" pod "dc86d00f-2713-4d37-993d-63e87d23f495" (UID: "dc86d00f-2713-4d37-993d-63e87d23f495"). InnerVolumeSpecName "kube-api-access-l4xnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:57.198286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.198252 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np"] Nov 02 15:21:57.199067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.199039 5476 cpu_manager.go:414] "containerMap: removing container" podUID="de02b6ea-99c6-4047-827a-909cc2d60367" containerName="revoke-keys" Nov 02 15:21:57.199067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.199057 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de02b6ea-99c6-4047-827a-909cc2d60367" containerName="revoke-keys" Nov 02 15:21:57.199207 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.199086 5476 cpu_manager.go:414] "containerMap: removing container" podUID="dc86d00f-2713-4d37-993d-63e87d23f495" containerName="revoke-keys" Nov 02 15:21:57.199207 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.199091 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc86d00f-2713-4d37-993d-63e87d23f495" containerName="revoke-keys" Nov 02 15:21:57.199207 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.199189 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="de02b6ea-99c6-4047-827a-909cc2d60367" containerName="revoke-keys" Nov 02 15:21:57.199207 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.199201 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc86d00f-2713-4d37-993d-63e87d23f495" containerName="revoke-keys" Nov 02 15:21:57.203013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.202977 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:21:57.227051 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.227023 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4xnd\" (UniqueName: \"kubernetes.io/projected/dc86d00f-2713-4d37-993d-63e87d23f495-kube-api-access-l4xnd\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:57.227159 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.227054 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc86d00f-2713-4d37-993d-63e87d23f495-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:57.258177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.258151 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np"] Nov 02 15:21:57.328227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.328188 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f7c3da-rs9np\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:21:57.328403 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.328385 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f7c3da-rs9np\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:21:57.328451 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.328408 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8dqm\" (UniqueName: \"kubernetes.io/projected/cf52a02e-c0fa-4351-89a9-7027069f4842-kube-api-access-j8dqm\") pod \"maas-api-revoke-sub-36f7c3da-rs9np\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:21:57.429759 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.429680 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f7c3da-rs9np\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:21:57.429903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.429850 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f7c3da-rs9np\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:21:57.429903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.429882 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8dqm\" (UniqueName: \"kubernetes.io/projected/cf52a02e-c0fa-4351-89a9-7027069f4842-kube-api-access-j8dqm\") pod \"maas-api-revoke-sub-36f7c3da-rs9np\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:21:57.430477 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.430453 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f7c3da-rs9np\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:21:57.432208 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.432179 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f7c3da-rs9np\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:21:57.488045 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.488011 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8dqm\" (UniqueName: \"kubernetes.io/projected/cf52a02e-c0fa-4351-89a9-7027069f4842-kube-api-access-j8dqm\") pod \"maas-api-revoke-sub-36f7c3da-rs9np\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:21:57.518091 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.518054 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:21:57.878883 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.878847 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" Nov 02 15:21:57.878883 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.878868 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds" event={"ID":"dc86d00f-2713-4d37-993d-63e87d23f495","Type":"ContainerDied","Data":"c85bf0ff3703bbfe01e8ed370ae90ef3e9093ae643aadf426ce9568f97e4a25e"} Nov 02 15:21:57.879358 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.878926 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c85bf0ff3703bbfe01e8ed370ae90ef3e9093ae643aadf426ce9568f97e4a25e" Nov 02 15:21:57.900217 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:57.900182 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np"] Nov 02 15:21:57.903446 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:21:57.903416 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf52a02e_c0fa_4351_89a9_7027069f4842.slice/crio-81659c56223fed6cf31b24267e000f2d5a0b1b17d2abfcd7404218fd7de1a619 WatchSource:0}: Error finding container 81659c56223fed6cf31b24267e000f2d5a0b1b17d2abfcd7404218fd7de1a619: Status 404 returned error can't find the container with id 81659c56223fed6cf31b24267e000f2d5a0b1b17d2abfcd7404218fd7de1a619 Nov 02 15:21:58.052020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:58.051960 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t"] Nov 02 15:21:58.056296 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:58.056267 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d469335-t964t"] Nov 02 15:21:58.890548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:58.890514 5476 generic.go:356] "Generic (PLEG): container finished" podID="cf52a02e-c0fa-4351-89a9-7027069f4842" containerID="c0ec3e97d8f92bda094a9db04edafd9d9502896456ae51811c029a2af9c64572" exitCode=0 Nov 02 15:21:58.891094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:58.890593 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" event={"ID":"cf52a02e-c0fa-4351-89a9-7027069f4842","Type":"ContainerDied","Data":"c0ec3e97d8f92bda094a9db04edafd9d9502896456ae51811c029a2af9c64572"} Nov 02 15:21:58.891094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:58.890677 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" event={"ID":"cf52a02e-c0fa-4351-89a9-7027069f4842","Type":"ContainerStarted","Data":"81659c56223fed6cf31b24267e000f2d5a0b1b17d2abfcd7404218fd7de1a619"} Nov 02 15:21:59.184263 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.184236 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:21:59.255742 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.255700 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-envoy\") pod \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " Nov 02 15:21:59.255742 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.255749 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-data\") pod \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " Nov 02 15:21:59.256075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.255792 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istiod-ca-cert\") pod \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " Nov 02 15:21:59.256075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.255908 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-kube-api-access-8wd4q\" (UniqueName: \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-kube-api-access-8wd4q\") pod \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " Nov 02 15:21:59.256075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.255964 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-credential-socket\") pod \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " Nov 02 15:21:59.256075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256048 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-certs\") pod \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " Nov 02 15:21:59.256300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256125 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-podinfo\") pod \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " Nov 02 15:21:59.256300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256112 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-data" pod "f019ab4f-b2a6-458c-946c-1ec06e443ed7" (UID: "f019ab4f-b2a6-458c-946c-1ec06e443ed7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.256300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256175 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-socket\") pod \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " Nov 02 15:21:59.256300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256262 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-token\" (UniqueName: \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-token\") pod \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\" (UID: \"f019ab4f-b2a6-458c-946c-1ec06e443ed7\") " Nov 02 15:21:59.256300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256278 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-credential-socket" pod "f019ab4f-b2a6-458c-946c-1ec06e443ed7" (UID: "f019ab4f-b2a6-458c-946c-1ec06e443ed7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.256536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256392 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-certs" pod "f019ab4f-b2a6-458c-946c-1ec06e443ed7" (UID: "f019ab4f-b2a6-458c-946c-1ec06e443ed7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.256536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256307 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istiod-ca-cert" pod "f019ab4f-b2a6-458c-946c-1ec06e443ed7" (UID: "f019ab4f-b2a6-458c-946c-1ec06e443ed7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:59.256536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256491 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-socket" pod "f019ab4f-b2a6-458c-946c-1ec06e443ed7" (UID: "f019ab4f-b2a6-458c-946c-1ec06e443ed7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.256897 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256878 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.256897 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256899 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.257066 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256910 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.257066 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256919 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.257066 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.256927 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.258544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.258522 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-envoy" pod "f019ab4f-b2a6-458c-946c-1ec06e443ed7" (UID: "f019ab4f-b2a6-458c-946c-1ec06e443ed7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.258605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.258569 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-kube-api-access-8wd4q" pod "f019ab4f-b2a6-458c-946c-1ec06e443ed7" (UID: "f019ab4f-b2a6-458c-946c-1ec06e443ed7"). InnerVolumeSpecName "kube-api-access-8wd4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:59.258670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.258655 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-podinfo" pod "f019ab4f-b2a6-458c-946c-1ec06e443ed7" (UID: "f019ab4f-b2a6-458c-946c-1ec06e443ed7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:21:59.258837 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.258817 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-token" pod "f019ab4f-b2a6-458c-946c-1ec06e443ed7" (UID: "f019ab4f-b2a6-458c-946c-1ec06e443ed7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:59.358922 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.358879 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.358922 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.358910 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.358922 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.358920 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wd4q\" (UniqueName: \"kubernetes.io/projected/f019ab4f-b2a6-458c-946c-1ec06e443ed7-kube-api-access-8wd4q\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.358922 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.358932 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f019ab4f-b2a6-458c-946c-1ec06e443ed7-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.900378 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.900340 5476 generic.go:356] "Generic (PLEG): container finished" podID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerID="47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99" exitCode=0 Nov 02 15:21:59.900822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.900400 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" Nov 02 15:21:59.900822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.900426 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" event={"ID":"f019ab4f-b2a6-458c-946c-1ec06e443ed7","Type":"ContainerDied","Data":"47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99"} Nov 02 15:21:59.900822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.900483 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b" event={"ID":"f019ab4f-b2a6-458c-946c-1ec06e443ed7","Type":"ContainerDied","Data":"24131866a2bdf378c0311773d89863a90b6e58852113578f42e1a6eabbb85af0"} Nov 02 15:21:59.900822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.900505 5476 scope.go:122] "RemoveContainer" containerID="47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99" Nov 02 15:21:59.913419 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.913390 5476 scope.go:122] "RemoveContainer" containerID="47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99" Nov 02 15:21:59.913734 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:21:59.913714 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99\": container with ID starting with 47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99 not found: ID does not exist" containerID="47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99" Nov 02 15:21:59.913810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.913749 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99"} err="failed to get container status \"47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99\": rpc error: code = NotFound desc = could not find container \"47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99\": container with ID starting with 47996c9bc3546b6c0a4fd67c6797550b6ae85788efb54a6f8c5fcead05cfcd99 not found: ID does not exist" Nov 02 15:21:59.924376 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.924348 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcf1cd02-304f-4482-9e8b-94d08701aceb" path="/var/lib/kubelet/pods/fcf1cd02-304f-4482-9e8b-94d08701aceb/volumes" Nov 02 15:21:59.944154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.944115 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz"] Nov 02 15:21:59.945055 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.945032 5476 cpu_manager.go:414] "containerMap: removing container" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerName="istio-proxy" Nov 02 15:21:59.945055 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.945053 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerName="istio-proxy" Nov 02 15:21:59.945225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.945181 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" containerName="istio-proxy" Nov 02 15:21:59.950355 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.950336 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:21:59.968041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.967131 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:21:59.968041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.967200 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:21:59.968041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.967248 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:21:59.968041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.967300 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:21:59.968041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.967343 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv2xs\" (UniqueName: \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-kube-api-access-dv2xs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:21:59.968041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.967384 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:21:59.968041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.967471 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04768429-0d60-4444-a82c-e6642e2445d6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:21:59.999080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.999042 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 15:21:59.999265 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.999149 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 15:21:59.999265 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.999149 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 15:21:59.999265 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:21:59.999197 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wfw99\"" Nov 02 15:22:00.011079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.011045 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz"] Nov 02 15:22:00.058266 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.058229 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:22:00.060078 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.060051 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b"] Nov 02 15:22:00.064348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.064324 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-zz92b"] Nov 02 15:22:00.068067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.068047 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-xks-ca\") pod \"cf52a02e-c0fa-4351-89a9-7027069f4842\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " Nov 02 15:22:00.068164 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.068109 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf52a02e-c0fa-4351-89a9-7027069f4842-kube-api-access-j8dqm\" (UniqueName: \"kubernetes.io/projected/cf52a02e-c0fa-4351-89a9-7027069f4842-kube-api-access-j8dqm\") pod \"cf52a02e-c0fa-4351-89a9-7027069f4842\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " Nov 02 15:22:00.068212 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.068197 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-openshift-ca\") pod \"cf52a02e-c0fa-4351-89a9-7027069f4842\" (UID: \"cf52a02e-c0fa-4351-89a9-7027069f4842\") " Nov 02 15:22:00.068577 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.068556 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-openshift-ca" pod "cf52a02e-c0fa-4351-89a9-7027069f4842" (UID: "cf52a02e-c0fa-4351-89a9-7027069f4842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:22:00.068643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.068599 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.068698 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.068657 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.068698 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.068692 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.068808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.068713 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.068808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.068733 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv2xs\" (UniqueName: \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-kube-api-access-dv2xs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.068808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.068778 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.068964 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.068843 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04768429-0d60-4444-a82c-e6642e2445d6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.069082 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.069063 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:00.070132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.070104 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.070691 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.070662 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf52a02e-c0fa-4351-89a9-7027069f4842-kube-api-access-j8dqm" pod "cf52a02e-c0fa-4351-89a9-7027069f4842" (UID: "cf52a02e-c0fa-4351-89a9-7027069f4842"). InnerVolumeSpecName "kube-api-access-j8dqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:00.070781 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.070749 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-xks-ca" pod "cf52a02e-c0fa-4351-89a9-7027069f4842" (UID: "cf52a02e-c0fa-4351-89a9-7027069f4842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:22:00.071293 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.071263 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.071371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.071309 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.071371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.071350 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.071707 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.071691 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04768429-0d60-4444-a82c-e6642e2445d6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.126748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.126717 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv2xs\" (UniqueName: \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-kube-api-access-dv2xs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.126898 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.126749 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-z7hdz\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.170785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.170757 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf52a02e-c0fa-4351-89a9-7027069f4842-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:00.170785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.170784 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8dqm\" (UniqueName: \"kubernetes.io/projected/cf52a02e-c0fa-4351-89a9-7027069f4842-kube-api-access-j8dqm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:00.264028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.263984 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:00.722392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.722319 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz"] Nov 02 15:22:00.725027 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:22:00.724966 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04768429_0d60_4444_a82c_e6642e2445d6.slice/crio-9a3a8e55fc1ae4b4abc51057f0211a1ae1ef456fae677984d6e725a26b808aed WatchSource:0}: Error finding container 9a3a8e55fc1ae4b4abc51057f0211a1ae1ef456fae677984d6e725a26b808aed: Status 404 returned error can't find the container with id 9a3a8e55fc1ae4b4abc51057f0211a1ae1ef456fae677984d6e725a26b808aed Nov 02 15:22:00.727247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.727228 5476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:22:00.910282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.910251 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" Nov 02 15:22:00.910763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.910277 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np" event={"ID":"cf52a02e-c0fa-4351-89a9-7027069f4842","Type":"ContainerDied","Data":"81659c56223fed6cf31b24267e000f2d5a0b1b17d2abfcd7404218fd7de1a619"} Nov 02 15:22:00.910763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.910354 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81659c56223fed6cf31b24267e000f2d5a0b1b17d2abfcd7404218fd7de1a619" Nov 02 15:22:00.912009 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:00.911964 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" event={"ID":"04768429-0d60-4444-a82c-e6642e2445d6","Type":"ContainerStarted","Data":"9a3a8e55fc1ae4b4abc51057f0211a1ae1ef456fae677984d6e725a26b808aed"} Nov 02 15:22:01.925112 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:01.925071 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f019ab4f-b2a6-458c-946c-1ec06e443ed7" path="/var/lib/kubelet/pods/f019ab4f-b2a6-458c-946c-1ec06e443ed7/volumes" Nov 02 15:22:04.684693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:04.684660 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:04.684983 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:04.684729 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:04.951360 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:04.951258 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" event={"ID":"04768429-0d60-4444-a82c-e6642e2445d6","Type":"ContainerStarted","Data":"c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353"} Nov 02 15:22:04.951360 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:04.951344 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:05.156864 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:05.156814 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" podStartSLOduration=2.200935589 podStartE2EDuration="6.156801471s" totalImagesPullingTime="3.955865882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:05.110648466 +0000 UTC m=+1427.313526517" watchObservedRunningTime="2026-11-02 15:22:05.156801471 +0000 UTC m=+1427.359679524" Nov 02 15:22:05.980013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:05.973619 5476 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-z7hdz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:22:05.980013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:05.973690 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" podUID="04768429-0d60-4444-a82c-e6642e2445d6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:22:06.083075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.083039 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z"] Nov 02 15:22:06.084300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.084269 5476 cpu_manager.go:414] "containerMap: removing container" podUID="cf52a02e-c0fa-4351-89a9-7027069f4842" containerName="revoke-keys" Nov 02 15:22:06.084300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.084295 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf52a02e-c0fa-4351-89a9-7027069f4842" containerName="revoke-keys" Nov 02 15:22:06.084501 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.084461 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf52a02e-c0fa-4351-89a9-7027069f4842" containerName="revoke-keys" Nov 02 15:22:06.089423 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.089398 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:06.138723 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.138682 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:22:06.144104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.144072 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z"] Nov 02 15:22:06.177337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.177230 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f23e94e-zzv4z\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:06.177337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.177281 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknrl\" (UniqueName: \"kubernetes.io/projected/7ed99465-e804-4ae5-8237-67888b99672e-kube-api-access-kknrl\") pod \"maas-api-revoke-sub-9f23e94e-zzv4z\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:06.177566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.177495 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f23e94e-zzv4z\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:06.182918 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.182896 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:22:06.279343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.279247 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f23e94e-zzv4z\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:06.279343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.279298 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknrl\" (UniqueName: \"kubernetes.io/projected/7ed99465-e804-4ae5-8237-67888b99672e-kube-api-access-kknrl\") pod \"maas-api-revoke-sub-9f23e94e-zzv4z\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:06.279569 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.279437 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f23e94e-zzv4z\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:06.280144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.280113 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f23e94e-zzv4z\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:06.281942 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.281916 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f23e94e-zzv4z\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:06.339389 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.339359 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknrl\" (UniqueName: \"kubernetes.io/projected/7ed99465-e804-4ae5-8237-67888b99672e-kube-api-access-kknrl\") pod \"maas-api-revoke-sub-9f23e94e-zzv4z\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:06.406166 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.406121 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:06.785217 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.785193 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z"] Nov 02 15:22:06.788471 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:22:06.788441 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed99465_e804_4ae5_8237_67888b99672e.slice/crio-fa343902bec88138b83cff50fc935e14d7f0e59c2d735ec4390c561e51620ed5 WatchSource:0}: Error finding container fa343902bec88138b83cff50fc935e14d7f0e59c2d735ec4390c561e51620ed5: Status 404 returned error can't find the container with id fa343902bec88138b83cff50fc935e14d7f0e59c2d735ec4390c561e51620ed5 Nov 02 15:22:06.974537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:06.974499 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" event={"ID":"7ed99465-e804-4ae5-8237-67888b99672e","Type":"ContainerStarted","Data":"fa343902bec88138b83cff50fc935e14d7f0e59c2d735ec4390c561e51620ed5"} Nov 02 15:22:07.987732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:07.987645 5476 generic.go:356] "Generic (PLEG): container finished" podID="7ed99465-e804-4ae5-8237-67888b99672e" containerID="6c944e37da924a1d7e5e18f8b15ff4afe401e322b42e9518ac786b9f59c1d1f5" exitCode=0 Nov 02 15:22:07.988138 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:07.987733 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" event={"ID":"7ed99465-e804-4ae5-8237-67888b99672e","Type":"ContainerDied","Data":"6c944e37da924a1d7e5e18f8b15ff4afe401e322b42e9518ac786b9f59c1d1f5"} Nov 02 15:22:08.966415 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:08.966387 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:22:09.187144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:09.187111 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:09.325316 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:09.325239 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-xks-ca\") pod \"7ed99465-e804-4ae5-8237-67888b99672e\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " Nov 02 15:22:09.325483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:09.325385 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed99465-e804-4ae5-8237-67888b99672e-kube-api-access-kknrl\" (UniqueName: \"kubernetes.io/projected/7ed99465-e804-4ae5-8237-67888b99672e-kube-api-access-kknrl\") pod \"7ed99465-e804-4ae5-8237-67888b99672e\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " Nov 02 15:22:09.325483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:09.325428 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-openshift-ca\") pod \"7ed99465-e804-4ae5-8237-67888b99672e\" (UID: \"7ed99465-e804-4ae5-8237-67888b99672e\") " Nov 02 15:22:09.325902 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:09.325869 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-openshift-ca" pod "7ed99465-e804-4ae5-8237-67888b99672e" (UID: "7ed99465-e804-4ae5-8237-67888b99672e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:22:09.326258 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:09.326233 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:09.328147 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:09.328120 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-xks-ca" pod "7ed99465-e804-4ae5-8237-67888b99672e" (UID: "7ed99465-e804-4ae5-8237-67888b99672e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:22:09.328388 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:09.328364 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed99465-e804-4ae5-8237-67888b99672e-kube-api-access-kknrl" pod "7ed99465-e804-4ae5-8237-67888b99672e" (UID: "7ed99465-e804-4ae5-8237-67888b99672e"). InnerVolumeSpecName "kube-api-access-kknrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:09.427159 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:09.427139 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ed99465-e804-4ae5-8237-67888b99672e-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:09.427159 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:09.427159 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kknrl\" (UniqueName: \"kubernetes.io/projected/7ed99465-e804-4ae5-8237-67888b99672e-kube-api-access-kknrl\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:10.005177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:10.005143 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" event={"ID":"7ed99465-e804-4ae5-8237-67888b99672e","Type":"ContainerDied","Data":"fa343902bec88138b83cff50fc935e14d7f0e59c2d735ec4390c561e51620ed5"} Nov 02 15:22:10.005340 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:10.005191 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa343902bec88138b83cff50fc935e14d7f0e59c2d735ec4390c561e51620ed5" Nov 02 15:22:10.005340 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:10.005202 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z" Nov 02 15:22:13.049651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:13.049620 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9"] Nov 02 15:22:13.095683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:13.095654 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63294e77-5srt9"] Nov 02 15:22:13.924053 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:13.923980 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84534f8e-bbbe-4660-b117-9b0025a3d6ed" path="/var/lib/kubelet/pods/84534f8e-bbbe-4660-b117-9b0025a3d6ed/volumes" Nov 02 15:22:18.168710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:18.168676 5476 scope.go:122] "RemoveContainer" containerID="e18634d7b06434c720941b4d72c0aea3f378924fa4599add6a88ee5728dc1f82" Nov 02 15:22:18.178915 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:18.178896 5476 scope.go:122] "RemoveContainer" containerID="257c8e55835083bb62785d47557fa3d588b0a4eee937294b80eef44e56bf27fb" Nov 02 15:22:18.188332 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:18.188312 5476 scope.go:122] "RemoveContainer" containerID="ce918191603446d8fea4a965b0b4185cf7767a6c9d4cf25bcfdee97ee565ffb0" Nov 02 15:22:18.197817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:18.197795 5476 scope.go:122] "RemoveContainer" containerID="1057ded6d91bd30a6aa00ccd5c8b39e295a856594a12f60c38e03111841288b1" Nov 02 15:22:18.207593 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:18.207575 5476 scope.go:122] "RemoveContainer" containerID="040ff0dbb31d484299bf9ec79d70f59fd2060c81ac0607144589a1699963953f" Nov 02 15:22:18.216877 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:18.216860 5476 scope.go:122] "RemoveContainer" containerID="5bb17196b4f661e87af46fa18157be6677581a3697072c5f06147090942ccbf0" Nov 02 15:22:20.148644 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:22:20.148604 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:22:20.294760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:20.294733 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:22:22.273285 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:22.271436 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:22.281642 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:22.281585 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:22.706749 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:22.706702 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:22.800018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:22.763630 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:22.853015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:22.827532 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:22.857365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:22.856273 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:22.862023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:22.858035 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.075107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.068195 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:22:23.075107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.068275 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.125360 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.125315 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 15:22:23.126042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.125593 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.126917 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.125635 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:22:23.276840 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.271710 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.276840 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.272171 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:22:23.295046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.292112 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:22:23.295046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.293802 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:22:23.307593 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.307255 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.664343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.664298 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.787173 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.782153 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 15:22:23.787173 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.782877 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:22:23.787173 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.784113 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 15:22:23.787173 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.784411 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:22:23.917308 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.907265 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:22:23.939866 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:23.939803 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.148016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.144409 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.472020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.469108 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:22:24.472020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.469407 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:22:24.522038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.499334 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:22:24.543792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.543752 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:22:24.571293 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.571242 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.571636 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.571604 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:22:24.571816 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.571794 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:22:24.753072 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.625964 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 15:22:24.799071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.753708 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:22:24.907046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.897662 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 15:22:24.913139 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.907336 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.913139 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.909200 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.954028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:24.953967 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 02 15:22:25.208251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.194508 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 15:22:25.370045 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.361784 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:22:25.370373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.370351 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:22:25.455179 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.455151 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:22:25.492077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.459029 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:22:25.560173 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.560125 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.560717 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.560684 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 02 15:22:25.626144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.626095 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.626836 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.626392 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:22:25.749099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.626430 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:22:25.749099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.626472 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.782017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.758239 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.843020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.837226 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.843390 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.843348 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-4zkcl" podUID="6abac5be-31c8-4f53-a556-cf43374c83a9" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.909139 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.909088 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Nov 02 15:22:25.974022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.910187 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.020116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:25.964787 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.073740 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.073693 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/health\": context deadline exceeded" Nov 02 15:22:26.104051 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.074867 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.214363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.214304 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 15:22:26.283226 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.283129 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Nov 02 15:22:26.296134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.296088 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.589186 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.589063 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/health\": context deadline exceeded" Nov 02 15:22:26.669362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.594198 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:22:26.670410 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.637751 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 02 15:22:26.670686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.655695 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:22:26.681931 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.671255 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:22:26.681931 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.671307 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.831137 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.831083 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.864420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.852124 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:22:26.909136 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.909088 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:22:26.960655 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.910552 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.998191 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:26.998146 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:27.027018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.002627 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Nov 02 15:22:27.027355 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.027306 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:22:27.143774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.143108 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 02 15:22:27.230022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.223155 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:27.448143 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.448048 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:22:27.490012 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.486579 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:22:27.524469 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.524443 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:22:27.639716 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.630304 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:22:27.640481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.640460 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:22:27.674554 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.668679 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:22:27.674554 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.671126 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:22:27.710057 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.708988 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:22:27.716956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.716928 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:22:27.733529 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:27.733492 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:22:44.482122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.482081 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2"] Nov 02 15:22:44.485538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.482917 5476 cpu_manager.go:414] "containerMap: removing container" podUID="7ed99465-e804-4ae5-8237-67888b99672e" containerName="revoke-keys" Nov 02 15:22:44.485538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.482941 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed99465-e804-4ae5-8237-67888b99672e" containerName="revoke-keys" Nov 02 15:22:44.485538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.483125 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed99465-e804-4ae5-8237-67888b99672e" containerName="revoke-keys" Nov 02 15:22:44.487716 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.487696 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.607074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.607042 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.607235 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.607103 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.607298 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.607228 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93564a79-a336-4637-8182-ce90ca9b9938-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.607298 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.607279 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.607424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.607398 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bx8\" (UniqueName: \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-kube-api-access-q2bx8\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.607489 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.607456 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.607609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.607591 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93564a79-a336-4637-8182-ce90ca9b9938-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.607673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.607661 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.607714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.607687 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.691231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.691196 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2"] Nov 02 15:22:44.709357 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.709328 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93564a79-a336-4637-8182-ce90ca9b9938-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.709525 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.709389 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.709525 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.709419 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.709649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.709558 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.709649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.709638 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.709763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.709700 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93564a79-a336-4637-8182-ce90ca9b9938-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.709763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.709731 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.709920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.709779 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bx8\" (UniqueName: \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-kube-api-access-q2bx8\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.709920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.709819 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.709920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.709830 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.710097 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.710026 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.710302 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.710255 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93564a79-a336-4637-8182-ce90ca9b9938-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.710640 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.710617 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.710856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.710833 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.712558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.712526 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.712854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.712830 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93564a79-a336-4637-8182-ce90ca9b9938-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.766970 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.766907 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.767141 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.767123 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bx8\" (UniqueName: \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-kube-api-access-q2bx8\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:44.803151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:44.803121 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:45.245605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:45.245572 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2"] Nov 02 15:22:45.249045 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:22:45.249018 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93564a79_a336_4637_8182_ce90ca9b9938.slice/crio-df9552317a27f5ddc74981aad4d1a3a2daa67fabf4a3fb25a9983ff427a18e94 WatchSource:0}: Error finding container df9552317a27f5ddc74981aad4d1a3a2daa67fabf4a3fb25a9983ff427a18e94: Status 404 returned error can't find the container with id df9552317a27f5ddc74981aad4d1a3a2daa67fabf4a3fb25a9983ff427a18e94 Nov 02 15:22:45.251381 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:45.251337 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.251445 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:45.251404 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.251445 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:45.251440 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.752061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:45.752029 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" event={"ID":"93564a79-a336-4637-8182-ce90ca9b9938","Type":"ContainerStarted","Data":"294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5"} Nov 02 15:22:45.752470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:45.752091 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" event={"ID":"93564a79-a336-4637-8182-ce90ca9b9938","Type":"ContainerStarted","Data":"df9552317a27f5ddc74981aad4d1a3a2daa67fabf4a3fb25a9983ff427a18e94"} Nov 02 15:22:45.804115 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:45.804079 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:45.868910 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:45.868860 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" podStartSLOduration=1.868844529 podStartE2EDuration="1.868844529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:45.867713905 +0000 UTC m=+1468.070591965" watchObservedRunningTime="2026-11-02 15:22:45.868844529 +0000 UTC m=+1468.071722583" Nov 02 15:22:46.806156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:46.806122 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 15:22:46.806547 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:46.806175 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" podUID="93564a79-a336-4637-8182-ce90ca9b9938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 15:22:47.804097 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:47.804064 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 15:22:47.804276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:47.804117 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" podUID="93564a79-a336-4637-8182-ce90ca9b9938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 15:22:48.803871 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:48.803833 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 15:22:48.804417 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:48.803888 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" podUID="93564a79-a336-4637-8182-ce90ca9b9938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 15:22:49.803673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:49.803636 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 15:22:49.803863 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:49.803714 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" podUID="93564a79-a336-4637-8182-ce90ca9b9938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 15:22:50.148497 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:22:50.148457 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:22:50.803582 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:50.803544 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 15:22:50.803814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:50.803625 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" podUID="93564a79-a336-4637-8182-ce90ca9b9938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 15:22:51.804432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:51.804403 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 15:22:51.804826 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:51.804460 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" podUID="93564a79-a336-4637-8182-ce90ca9b9938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 15:22:52.807881 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:52.807852 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:52.808312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:52.808296 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:22:52.808894 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:22:52.808876 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:23:04.182786 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.182711 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf"] Nov 02 15:23:04.188882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.188856 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:23:04.251748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.251694 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88dv\" (UniqueName: \"kubernetes.io/projected/6f0dccdd-426e-40a4-b92e-6ab31b55a660-kube-api-access-z88dv\") pod \"payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf\" (UID: \"6f0dccdd-426e-40a4-b92e-6ab31b55a660\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:23:04.251921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.251828 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0dccdd-426e-40a4-b92e-6ab31b55a660-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf\" (UID: \"6f0dccdd-426e-40a4-b92e-6ab31b55a660\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:23:04.292273 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.292237 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-89017f75\"" Nov 02 15:23:04.292492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.292287 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-89017f75-dockercfg-wt64f\"" Nov 02 15:23:04.306926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.306878 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf"] Nov 02 15:23:04.353841 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.353801 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0dccdd-426e-40a4-b92e-6ab31b55a660-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf\" (UID: \"6f0dccdd-426e-40a4-b92e-6ab31b55a660\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:23:04.354455 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.354424 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88dv\" (UniqueName: \"kubernetes.io/projected/6f0dccdd-426e-40a4-b92e-6ab31b55a660-kube-api-access-z88dv\") pod \"payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf\" (UID: \"6f0dccdd-426e-40a4-b92e-6ab31b55a660\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:23:04.357145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.357118 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0dccdd-426e-40a4-b92e-6ab31b55a660-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf\" (UID: \"6f0dccdd-426e-40a4-b92e-6ab31b55a660\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:23:04.412966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.412375 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88dv\" (UniqueName: \"kubernetes.io/projected/6f0dccdd-426e-40a4-b92e-6ab31b55a660-kube-api-access-z88dv\") pod \"payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf\" (UID: \"6f0dccdd-426e-40a4-b92e-6ab31b55a660\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:23:04.527639 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.527548 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:23:04.929411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.929385 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf"] Nov 02 15:23:04.932559 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:23:04.932533 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0dccdd_426e_40a4_b92e_6ab31b55a660.slice/crio-0e7c778544ed88d002c281415240edaba4a5c6fbbae65b604a9d03a77e6ea2d7 WatchSource:0}: Error finding container 0e7c778544ed88d002c281415240edaba4a5c6fbbae65b604a9d03a77e6ea2d7: Status 404 returned error can't find the container with id 0e7c778544ed88d002c281415240edaba4a5c6fbbae65b604a9d03a77e6ea2d7 Nov 02 15:23:04.982758 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:04.982732 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" event={"ID":"6f0dccdd-426e-40a4-b92e-6ab31b55a660","Type":"ContainerStarted","Data":"0e7c778544ed88d002c281415240edaba4a5c6fbbae65b604a9d03a77e6ea2d7"} Nov 02 15:23:05.992234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:05.992188 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" event={"ID":"6f0dccdd-426e-40a4-b92e-6ab31b55a660","Type":"ContainerStarted","Data":"dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46"} Nov 02 15:23:05.992622 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:05.992271 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:23:06.104601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:06.104525 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" podStartSLOduration=2.104500815 podStartE2EDuration="2.104500815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:06.101256505 +0000 UTC m=+1488.304134591" watchObservedRunningTime="2026-11-02 15:23:06.104500815 +0000 UTC m=+1488.307378888" Nov 02 15:23:17.013025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:17.012981 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:23:18.145529 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:18.145497 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:23:18.152512 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:18.152493 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:23:18.277781 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:18.277753 5476 scope.go:122] "RemoveContainer" containerID="f032dae382975db295aaa022d4a9f28eb81f4051cb396149678b4efe618bb697" Nov 02 15:23:18.288709 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:18.288690 5476 scope.go:122] "RemoveContainer" containerID="64bb21e2b950178aefa4b79a8b759025d8da1391942408ea455589f1b0d988c3" Nov 02 15:23:18.301014 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:18.300975 5476 scope.go:122] "RemoveContainer" containerID="7bf595a2d82c050336c6d15ddeb773f5af5a3ea8f5e64c3d503c4de1948190f0" Nov 02 15:23:18.310502 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:18.310483 5476 scope.go:122] "RemoveContainer" containerID="2b33ca2bb768cb47b4e489ccde2494f830b84b886f99bcdb6a4346f4dbd44f86" Nov 02 15:23:18.319472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:18.319452 5476 scope.go:122] "RemoveContainer" containerID="f7a26b070af869ca9bc9fd6b0df5b17eea9d0a95e0d463967b1dc9919c94580d" Nov 02 15:23:18.328088 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:18.328072 5476 scope.go:122] "RemoveContainer" containerID="cf392f639a069ecddb29ad4468d098aaaca890ba63f273e39c9f9abb6fd96e0b" Nov 02 15:23:18.336644 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:18.336626 5476 scope.go:122] "RemoveContainer" containerID="847f5a2dc86485350422b96aa291b192f77d8f80d0d9a842aea9627fb531e5ef" Nov 02 15:23:18.345675 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:18.345661 5476 scope.go:122] "RemoveContainer" containerID="05cb445a491d06367dfd7d3b9b12bd3f7ff48213b9010bc69d5c6ff301d64e53" Nov 02 15:23:19.186960 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.186931 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc"] Nov 02 15:23:19.193209 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.193188 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.243966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.243944 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-bd9398b4-serving-cert\"" Nov 02 15:23:19.267758 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.267734 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnzmz\" (UniqueName: \"kubernetes.io/projected/6069c8d1-a8ea-480b-828c-850bfec695c8-kube-api-access-wnzmz\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.267903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.267867 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-maas-api-tls\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.267981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.267927 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6069c8d1-a8ea-480b-828c-850bfec695c8-openshift-service-ca\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.267981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.267969 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-metrics-certs\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.296699 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.296665 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc"] Nov 02 15:23:19.369113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.369083 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-maas-api-tls\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.369269 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.369138 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6069c8d1-a8ea-480b-828c-850bfec695c8-openshift-service-ca\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.369269 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.369168 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-metrics-certs\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.369269 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:23:19.369249 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-serving-cert: secret "maas-api-e2e-derive-bd9398b4-serving-cert" not found Nov 02 15:23:19.369372 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:23:19.369319 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-maas-api-tls podName:6069c8d1-a8ea-480b-828c-850bfec695c8 nodeName:}" failed. No retries permitted until 2026-11-02 15:23:19.869298382 +0000 UTC m=+1502.072176442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-maas-api-tls") pod "maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" (UID: "6069c8d1-a8ea-480b-828c-850bfec695c8") : secret "maas-api-e2e-derive-bd9398b4-serving-cert" not found Nov 02 15:23:19.369454 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.369435 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnzmz\" (UniqueName: \"kubernetes.io/projected/6069c8d1-a8ea-480b-828c-850bfec695c8-kube-api-access-wnzmz\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.369858 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.369836 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6069c8d1-a8ea-480b-828c-850bfec695c8-openshift-service-ca\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.371520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.371496 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-metrics-certs\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.428782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.428760 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnzmz\" (UniqueName: \"kubernetes.io/projected/6069c8d1-a8ea-480b-828c-850bfec695c8-kube-api-access-wnzmz\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.878005 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.877961 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-maas-api-tls\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:19.880514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:19.880339 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-maas-api-tls\") pod \"maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:20.108152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:20.108122 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:20.150841 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:23:20.150726 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:23:20.493505 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:20.493428 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc"] Nov 02 15:23:20.497311 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:23:20.497283 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6069c8d1_a8ea_480b_828c_850bfec695c8.slice/crio-59e08355760efedf3968186cb75b002592878c15071caee90e360ebe717150c2 WatchSource:0}: Error finding container 59e08355760efedf3968186cb75b002592878c15071caee90e360ebe717150c2: Status 404 returned error can't find the container with id 59e08355760efedf3968186cb75b002592878c15071caee90e360ebe717150c2 Nov 02 15:23:21.151779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:21.151744 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" event={"ID":"6069c8d1-a8ea-480b-828c-850bfec695c8","Type":"ContainerStarted","Data":"59e08355760efedf3968186cb75b002592878c15071caee90e360ebe717150c2"} Nov 02 15:23:22.162083 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:22.162049 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" event={"ID":"6069c8d1-a8ea-480b-828c-850bfec695c8","Type":"ContainerStarted","Data":"846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6"} Nov 02 15:23:22.162469 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:22.162141 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:22.277312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:22.277267 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" podStartSLOduration=2.397531193 podStartE2EDuration="3.27725259s" totalImagesPullingTime="879.721397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:22.271646695 +0000 UTC m=+1504.474524755" watchObservedRunningTime="2026-11-02 15:23:22.27725259 +0000 UTC m=+1504.480130678" Nov 02 15:23:28.175745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:28.175711 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:23:29.744119 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:29.744088 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:23:31.936036 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:31.935977 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:31.942118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:31.942087 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.518254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:32.518209 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:23:32.678216 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:32.678164 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.718126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:32.718079 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.730095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:32.730054 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:23:32.731128 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:32.731097 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.849201 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:32.849076 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.922099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:32.922061 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.938685 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:32.938655 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.939641 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:32.939621 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:23:33.523879 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:33.523371 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded" Nov 02 15:23:33.527601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:33.527567 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:23:33.562684 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:33.562641 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:23:33.583149 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:23:33.583121 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:23:50.147686 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:23:50.147650 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:24:02.041520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.041482 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc"] Nov 02 15:24:02.304171 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.304076 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc"] Nov 02 15:24:02.304524 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.304503 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.358810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.358770 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-89017f75-serving-cert\"" Nov 02 15:24:02.460201 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.460158 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-metrics-certs\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.460581 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.460555 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6q5x\" (UniqueName: \"kubernetes.io/projected/6056636e-d5c8-4b41-ac22-90cfa67a9b87-kube-api-access-z6q5x\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.460788 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.460769 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6056636e-d5c8-4b41-ac22-90cfa67a9b87-openshift-service-ca\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.461095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.461075 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-maas-api-tls\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.627074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.619818 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-maas-api-tls\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.627074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.619922 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-metrics-certs\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.627074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.620110 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6q5x\" (UniqueName: \"kubernetes.io/projected/6056636e-d5c8-4b41-ac22-90cfa67a9b87-kube-api-access-z6q5x\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.627074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.620213 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6056636e-d5c8-4b41-ac22-90cfa67a9b87-openshift-service-ca\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.628137 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:24:02.628109 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-serving-cert: secret "maas-api-e2e-ait-89017f75-serving-cert" not found Nov 02 15:24:02.628378 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:24:02.628363 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-maas-api-tls podName:6056636e-d5c8-4b41-ac22-90cfa67a9b87 nodeName:}" failed. No retries permitted until 2026-11-02 15:24:03.128329131 +0000 UTC m=+1545.331207187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-maas-api-tls") pod "maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" (UID: "6056636e-d5c8-4b41-ac22-90cfa67a9b87") : secret "maas-api-e2e-ait-89017f75-serving-cert" not found Nov 02 15:24:02.649565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.649531 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6056636e-d5c8-4b41-ac22-90cfa67a9b87-openshift-service-ca\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.730019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.691310 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:02.785699 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.714630 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-metrics-certs\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.786629 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.730290 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:24:02.787541 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.731920 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6q5x\" (UniqueName: \"kubernetes.io/projected/6056636e-d5c8-4b41-ac22-90cfa67a9b87-kube-api-access-z6q5x\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:02.787541 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.732628 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:02.880754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.880654 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:02.942064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:02.942023 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:03.220350 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:03.219425 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-maas-api-tls\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:03.225021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:03.224969 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-maas-api-tls\") pod \"maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:03.490748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:03.490654 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:03.885100 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:03.885064 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc"] Nov 02 15:24:03.886171 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:24:03.886146 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6056636e_d5c8_4b41_ac22_90cfa67a9b87.slice/crio-a131e46129fc5c55302f8a25bda4215d6ed4918aabd97e3174b33e8748508a62 WatchSource:0}: Error finding container a131e46129fc5c55302f8a25bda4215d6ed4918aabd97e3174b33e8748508a62: Status 404 returned error can't find the container with id a131e46129fc5c55302f8a25bda4215d6ed4918aabd97e3174b33e8748508a62 Nov 02 15:24:03.968070 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:03.968040 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" event={"ID":"6056636e-d5c8-4b41-ac22-90cfa67a9b87","Type":"ContainerStarted","Data":"a131e46129fc5c55302f8a25bda4215d6ed4918aabd97e3174b33e8748508a62"} Nov 02 15:24:04.980237 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:04.980135 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" event={"ID":"6056636e-d5c8-4b41-ac22-90cfa67a9b87","Type":"ContainerStarted","Data":"4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812"} Nov 02 15:24:04.980612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:04.980239 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:05.096537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:05.096472 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" podStartSLOduration=3.260754676 podStartE2EDuration="4.096457303s" totalImagesPullingTime="835.702627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:05.092709712 +0000 UTC m=+1547.295587772" watchObservedRunningTime="2026-11-02 15:24:05.096457303 +0000 UTC m=+1547.299335362" Nov 02 15:24:10.027168 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.027135 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9"] Nov 02 15:24:10.031638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.031620 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.138026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.136466 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.138026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.136533 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.138026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.136569 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.138026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.136602 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.138026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.136751 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.138026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.136839 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.138026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.137012 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7q5n\" (UniqueName: \"kubernetes.io/projected/afd13c1e-9b91-4be5-92ed-ceea14f0203c-kube-api-access-j7q5n\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.138026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.137063 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.138026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.137109 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.145036 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.139160 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9"] Nov 02 15:24:10.238298 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238257 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.238298 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238300 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.238537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238332 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.238537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238356 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.238537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238509 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.238693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238601 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.238856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238760 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7q5n\" (UniqueName: \"kubernetes.io/projected/afd13c1e-9b91-4be5-92ed-ceea14f0203c-kube-api-access-j7q5n\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.238856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238815 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.238856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238830 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.238856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238759 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.239123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238873 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.239123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.238914 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.239238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.239146 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.239601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.239577 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.240699 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.240680 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.240799 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.240783 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.296430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.296365 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd13c1e-9b91-4be5-92ed-ceea14f0203c-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.296631 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.296608 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7q5n\" (UniqueName: \"kubernetes.io/projected/afd13c1e-9b91-4be5-92ed-ceea14f0203c-kube-api-access-j7q5n\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kk2m9\" (UID: \"afd13c1e-9b91-4be5-92ed-ceea14f0203c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.347158 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.347123 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:10.778067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.777961 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9"] Nov 02 15:24:10.783673 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:24:10.783642 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd13c1e_9b91_4be5_92ed_ceea14f0203c.slice/crio-e70a487443db2cedbbe23fb5e36582fab2bfd7007ed087bfd872820f67b5e8b1 WatchSource:0}: Error finding container e70a487443db2cedbbe23fb5e36582fab2bfd7007ed087bfd872820f67b5e8b1: Status 404 returned error can't find the container with id e70a487443db2cedbbe23fb5e36582fab2bfd7007ed087bfd872820f67b5e8b1 Nov 02 15:24:10.786502 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.786469 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:10.786609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.786548 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:10.786609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.786591 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:10.993026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:10.992974 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:24:11.032237 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:11.032143 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" event={"ID":"afd13c1e-9b91-4be5-92ed-ceea14f0203c","Type":"ContainerStarted","Data":"42712c49a48be24d07db7e2fdc48d4a3e6b070a00a065255dd708e1bb615f9fd"} Nov 02 15:24:11.032237 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:11.032213 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" event={"ID":"afd13c1e-9b91-4be5-92ed-ceea14f0203c","Type":"ContainerStarted","Data":"e70a487443db2cedbbe23fb5e36582fab2bfd7007ed087bfd872820f67b5e8b1"} Nov 02 15:24:11.218692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:11.218634 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podStartSLOduration=2.21862086 podStartE2EDuration="2.21862086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:11.217079539 +0000 UTC m=+1553.419957596" watchObservedRunningTime="2026-11-02 15:24:11.21862086 +0000 UTC m=+1553.421498919" Nov 02 15:24:11.348188 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:11.348096 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:11.349539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:11.349516 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kk2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:11.349642 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:11.349563 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podUID="afd13c1e-9b91-4be5-92ed-ceea14f0203c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 15:24:12.348062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:12.348030 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kk2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:12.348452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:12.348091 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podUID="afd13c1e-9b91-4be5-92ed-ceea14f0203c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 15:24:13.347606 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:13.347574 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kk2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:13.347813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:13.347641 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podUID="afd13c1e-9b91-4be5-92ed-ceea14f0203c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 15:24:14.051279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:14.051235 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k"] Nov 02 15:24:14.056653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:14.056625 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21c6f771-pkn4k"] Nov 02 15:24:14.347973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:14.347891 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kk2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:14.347973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:14.347945 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podUID="afd13c1e-9b91-4be5-92ed-ceea14f0203c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 15:24:15.348350 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:15.348315 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kk2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:15.348738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:15.348369 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podUID="afd13c1e-9b91-4be5-92ed-ceea14f0203c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 15:24:15.925296 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:15.925263 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed438674-a077-43f3-b36f-0c7911ad0b3e" path="/var/lib/kubelet/pods/ed438674-a077-43f3-b36f-0c7911ad0b3e/volumes" Nov 02 15:24:16.347856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:16.347771 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kk2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:16.347856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:16.347836 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podUID="afd13c1e-9b91-4be5-92ed-ceea14f0203c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 15:24:17.348119 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:17.348083 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kk2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:17.348507 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:17.348135 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podUID="afd13c1e-9b91-4be5-92ed-ceea14f0203c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 15:24:18.347701 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:18.347665 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kk2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:18.347873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:18.347726 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podUID="afd13c1e-9b91-4be5-92ed-ceea14f0203c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" Nov 02 15:24:19.594542 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:19.594500 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:19.595049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:19.594841 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:19.595843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:19.595822 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" Nov 02 15:24:20.147796 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:24:20.147762 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:24:40.968037 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:40.967975 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:24:43.053215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:43.053170 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct"] Nov 02 15:24:43.099098 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:43.099069 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608dea81-dtpct"] Nov 02 15:24:43.929641 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:43.929602 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c68892-f972-467d-beab-8db2a842bb8f" path="/var/lib/kubelet/pods/a7c68892-f972-467d-beab-8db2a842bb8f/volumes" Nov 02 15:24:50.150185 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:24:50.150126 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:24:54.051615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:54.051560 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4"] Nov 02 15:24:54.097331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:54.097301 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6518b1b3-6rsq4"] Nov 02 15:24:55.926287 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:24:55.926254 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3487923-57a5-49cd-a252-3dabedca8603" path="/var/lib/kubelet/pods/a3487923-57a5-49cd-a252-3dabedca8603/volumes" Nov 02 15:25:13.057586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:13.057551 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf"] Nov 02 15:25:13.104696 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:13.104658 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b175f4c-sjngf"] Nov 02 15:25:13.938215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:13.938177 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b628982-17fa-459b-ad3b-881869e3f95f" path="/var/lib/kubelet/pods/6b628982-17fa-459b-ad3b-881869e3f95f/volumes" Nov 02 15:25:18.426038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:18.425986 5476 scope.go:122] "RemoveContainer" containerID="04a5af85c3482050c92d763fd5b435cbedfc7c8dcff58129479d0d0b1efc732a" Nov 02 15:25:20.249102 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:25:20.249062 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:25:25.610402 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.610371 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648"] Nov 02 15:25:25.615338 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.615316 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.678712 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.678682 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gph\" (UniqueName: \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-kube-api-access-f4gph\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.678867 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.678760 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.678867 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.678800 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.678955 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.678919 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.679076 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.679059 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.679144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.679129 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.679232 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.679219 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.679286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.679264 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.679365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.679352 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.710950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.710912 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648"] Nov 02 15:25:25.780946 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.780910 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.781150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.780987 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.781150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.781088 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.781150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.781116 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gph\" (UniqueName: \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-kube-api-access-f4gph\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.781842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.781216 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.781842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.781241 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.781842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.781271 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.781842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.781321 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.781842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.781521 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.781842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.781640 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.782241 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.781968 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.782302 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.782253 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.782455 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.782433 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.782523 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.782487 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.783603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.783575 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.784220 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.784196 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.838528 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.838499 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.838927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.838905 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gph\" (UniqueName: \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-kube-api-access-f4gph\") pod \"maas-default-gateway-openshift-default-7f5667f86c-75648\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:25.928774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:25.928709 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:26.231244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.231170 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc"] Nov 02 15:25:26.231386 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.231362 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" podUID="6069c8d1-a8ea-480b-828c-850bfec695c8" containerName="maas-api" containerID="cri-o://846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6" gracePeriod=30 Nov 02 15:25:26.408186 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.408160 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648"] Nov 02 15:25:26.410443 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:25:26.410410 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eda8baf_c3cb_40ca_9acc_219484ff4dd7.slice/crio-fa2a2688638634b6362c89ca0c577e1323ff209637b8ec84f67cd0d398e75a96 WatchSource:0}: Error finding container fa2a2688638634b6362c89ca0c577e1323ff209637b8ec84f67cd0d398e75a96: Status 404 returned error can't find the container with id fa2a2688638634b6362c89ca0c577e1323ff209637b8ec84f67cd0d398e75a96 Nov 02 15:25:26.414434 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.413667 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:26.414434 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.413761 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:26.414434 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.413805 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:26.532819 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.532787 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:25:26.599880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.597197 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-metrics-certs\") pod \"6069c8d1-a8ea-480b-828c-850bfec695c8\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " Nov 02 15:25:26.599880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.597374 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6069c8d1-a8ea-480b-828c-850bfec695c8-kube-api-access-wnzmz\" (UniqueName: \"kubernetes.io/projected/6069c8d1-a8ea-480b-828c-850bfec695c8-kube-api-access-wnzmz\") pod \"6069c8d1-a8ea-480b-828c-850bfec695c8\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " Nov 02 15:25:26.599880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.597444 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-maas-api-tls\") pod \"6069c8d1-a8ea-480b-828c-850bfec695c8\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " Nov 02 15:25:26.599880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.597490 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6069c8d1-a8ea-480b-828c-850bfec695c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6069c8d1-a8ea-480b-828c-850bfec695c8-openshift-service-ca\") pod \"6069c8d1-a8ea-480b-828c-850bfec695c8\" (UID: \"6069c8d1-a8ea-480b-828c-850bfec695c8\") " Nov 02 15:25:26.599880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.597944 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6069c8d1-a8ea-480b-828c-850bfec695c8-openshift-service-ca" pod "6069c8d1-a8ea-480b-828c-850bfec695c8" (UID: "6069c8d1-a8ea-480b-828c-850bfec695c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:26.599880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.598256 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6069c8d1-a8ea-480b-828c-850bfec695c8-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:26.600352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.600135 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-maas-api-tls" pod "6069c8d1-a8ea-480b-828c-850bfec695c8" (UID: "6069c8d1-a8ea-480b-828c-850bfec695c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:25:26.600701 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.600466 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6069c8d1-a8ea-480b-828c-850bfec695c8-kube-api-access-wnzmz" pod "6069c8d1-a8ea-480b-828c-850bfec695c8" (UID: "6069c8d1-a8ea-480b-828c-850bfec695c8"). InnerVolumeSpecName "kube-api-access-wnzmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:26.603038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.603008 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-metrics-certs" pod "6069c8d1-a8ea-480b-828c-850bfec695c8" (UID: "6069c8d1-a8ea-480b-828c-850bfec695c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:25:26.704969 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.704931 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnzmz\" (UniqueName: \"kubernetes.io/projected/6069c8d1-a8ea-480b-828c-850bfec695c8-kube-api-access-wnzmz\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:26.704969 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.704969 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:26.705573 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.704986 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6069c8d1-a8ea-480b-828c-850bfec695c8-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:26.869577 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.869531 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" event={"ID":"5eda8baf-c3cb-40ca-9acc-219484ff4dd7","Type":"ContainerStarted","Data":"ec5ecba1676c54fcfa5f68b065d05a1b21382beaa3df696be4ef996eccc4babb"} Nov 02 15:25:26.869759 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.869608 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" event={"ID":"5eda8baf-c3cb-40ca-9acc-219484ff4dd7","Type":"ContainerStarted","Data":"fa2a2688638634b6362c89ca0c577e1323ff209637b8ec84f67cd0d398e75a96"} Nov 02 15:25:26.874073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.874037 5476 generic.go:356] "Generic (PLEG): container finished" podID="6069c8d1-a8ea-480b-828c-850bfec695c8" containerID="846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6" exitCode=0 Nov 02 15:25:26.874073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.874062 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" Nov 02 15:25:26.874311 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.874128 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" event={"ID":"6069c8d1-a8ea-480b-828c-850bfec695c8","Type":"ContainerDied","Data":"846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6"} Nov 02 15:25:26.874311 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.874180 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc" event={"ID":"6069c8d1-a8ea-480b-828c-850bfec695c8","Type":"ContainerDied","Data":"59e08355760efedf3968186cb75b002592878c15071caee90e360ebe717150c2"} Nov 02 15:25:26.874311 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.874206 5476 scope.go:122] "RemoveContainer" containerID="846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6" Nov 02 15:25:26.886694 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.886676 5476 scope.go:122] "RemoveContainer" containerID="846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6" Nov 02 15:25:26.887089 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:25:26.887062 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6\": container with ID starting with 846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6 not found: ID does not exist" containerID="846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6" Nov 02 15:25:26.887172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.887100 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6"} err="failed to get container status \"846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6\": rpc error: code = NotFound desc = could not find container \"846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6\": container with ID starting with 846206301f678ff1ee7abee86a6634caeb43d744d7fec333e4ca5d98b7e417d6 not found: ID does not exist" Nov 02 15:25:26.929650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.929620 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:26.986596 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:26.986489 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" podStartSLOduration=1.9864719530000001 podStartE2EDuration="1.986471953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:26.983886228 +0000 UTC m=+1629.186764313" watchObservedRunningTime="2026-11-02 15:25:26.986471953 +0000 UTC m=+1629.189350081" Nov 02 15:25:27.099519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:27.094811 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc"] Nov 02 15:25:27.106360 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:27.102181 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-bd9398b4-9bf88cf6f-d5flc"] Nov 02 15:25:27.925585 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:27.925515 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6069c8d1-a8ea-480b-828c-850bfec695c8" path="/var/lib/kubelet/pods/6069c8d1-a8ea-480b-828c-850bfec695c8/volumes" Nov 02 15:25:27.930946 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:27.930921 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-75648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:25:27.931151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:27.930966 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:25:28.946017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:28.945223 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-75648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:25:28.946017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:28.945285 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:25:29.050132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:29.050089 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58"] Nov 02 15:25:29.057501 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:29.057461 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc2dd041-j5l58"] Nov 02 15:25:29.924706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:29.924674 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41f318d-c9ac-4d0f-8da2-9fac8931e0ea" path="/var/lib/kubelet/pods/b41f318d-c9ac-4d0f-8da2-9fac8931e0ea/volumes" Nov 02 15:25:29.929115 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:29.929090 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-75648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:25:29.929259 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:29.929137 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:25:30.930059 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:30.930020 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-75648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:25:30.930521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:30.930080 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:25:31.216698 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:31.216609 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf"] Nov 02 15:25:31.929687 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:31.929652 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-75648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:25:31.929900 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:31.929767 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:25:32.930157 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:32.930123 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-75648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:25:32.930602 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:32.930186 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:25:33.929833 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:33.929804 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-75648 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 15:25:33.930021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:33.929852 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 15:25:35.576100 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:35.576058 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:35.576615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:35.576567 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:35.577173 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:35.577146 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:25:36.217854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:36.217815 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" podUID="6f0dccdd-426e-40a4-b92e-6ab31b55a660" containerName="payload-pre-processing" containerID="cri-o://dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46" gracePeriod=25 Nov 02 15:25:36.501085 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:36.501065 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:25:36.628492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:36.628458 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0dccdd-426e-40a4-b92e-6ab31b55a660-kube-api-access-z88dv\" (UniqueName: \"kubernetes.io/projected/6f0dccdd-426e-40a4-b92e-6ab31b55a660-kube-api-access-z88dv\") pod \"6f0dccdd-426e-40a4-b92e-6ab31b55a660\" (UID: \"6f0dccdd-426e-40a4-b92e-6ab31b55a660\") " Nov 02 15:25:36.628884 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:36.628558 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0dccdd-426e-40a4-b92e-6ab31b55a660-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0dccdd-426e-40a4-b92e-6ab31b55a660-plugins-config-volume\") pod \"6f0dccdd-426e-40a4-b92e-6ab31b55a660\" (UID: \"6f0dccdd-426e-40a4-b92e-6ab31b55a660\") " Nov 02 15:25:36.629005 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:36.628965 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0dccdd-426e-40a4-b92e-6ab31b55a660-plugins-config-volume" pod "6f0dccdd-426e-40a4-b92e-6ab31b55a660" (UID: "6f0dccdd-426e-40a4-b92e-6ab31b55a660"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:36.629342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:36.629315 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0dccdd-426e-40a4-b92e-6ab31b55a660-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:36.630844 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:36.630813 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0dccdd-426e-40a4-b92e-6ab31b55a660-kube-api-access-z88dv" pod "6f0dccdd-426e-40a4-b92e-6ab31b55a660" (UID: "6f0dccdd-426e-40a4-b92e-6ab31b55a660"). InnerVolumeSpecName "kube-api-access-z88dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:36.730977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:36.730884 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z88dv\" (UniqueName: \"kubernetes.io/projected/6f0dccdd-426e-40a4-b92e-6ab31b55a660-kube-api-access-z88dv\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:37.023444 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:37.023365 5476 generic.go:356] "Generic (PLEG): container finished" podID="6f0dccdd-426e-40a4-b92e-6ab31b55a660" containerID="dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46" exitCode=0 Nov 02 15:25:37.023444 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:37.023407 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" Nov 02 15:25:37.023674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:37.023452 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" event={"ID":"6f0dccdd-426e-40a4-b92e-6ab31b55a660","Type":"ContainerDied","Data":"dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46"} Nov 02 15:25:37.023674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:37.023513 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf" event={"ID":"6f0dccdd-426e-40a4-b92e-6ab31b55a660","Type":"ContainerDied","Data":"0e7c778544ed88d002c281415240edaba4a5c6fbbae65b604a9d03a77e6ea2d7"} Nov 02 15:25:37.023674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:37.023545 5476 scope.go:122] "RemoveContainer" containerID="dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46" Nov 02 15:25:37.042480 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:37.042453 5476 scope.go:122] "RemoveContainer" containerID="dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46" Nov 02 15:25:37.042817 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:25:37.042796 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46\": container with ID starting with dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46 not found: ID does not exist" containerID="dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46" Nov 02 15:25:37.042934 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:37.042823 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46"} err="failed to get container status \"dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46\": rpc error: code = NotFound desc = could not find container \"dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46\": container with ID starting with dafecdceaa8397f846832d5a03df5b70d8eda26e95080bdd8389106aaae3ef46 not found: ID does not exist" Nov 02 15:25:37.185555 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:37.185522 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf"] Nov 02 15:25:37.231227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:37.231185 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89017f75-556549dc75-fsmxf"] Nov 02 15:25:37.930873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:37.930843 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0dccdd-426e-40a4-b92e-6ab31b55a660" path="/var/lib/kubelet/pods/6f0dccdd-426e-40a4-b92e-6ab31b55a660/volumes" Nov 02 15:25:40.054850 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:40.054809 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f"] Nov 02 15:25:40.100357 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:40.100326 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6189297-jw45f"] Nov 02 15:25:41.925373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:41.925340 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe793fd-1322-4e57-b2c0-29b2ba95c35f" path="/var/lib/kubelet/pods/cfe793fd-1322-4e57-b2c0-29b2ba95c35f/volumes" Nov 02 15:25:45.053491 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:45.053454 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:25:50.148964 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:25:50.148919 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:25:53.056850 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:53.056815 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc"] Nov 02 15:25:53.102248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:53.102210 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc3ec49-c85qc"] Nov 02 15:25:53.925674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:53.925641 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5bbacd3-5f49-4704-a20b-ef15c4cc904c" path="/var/lib/kubelet/pods/a5bbacd3-5f49-4704-a20b-ef15c4cc904c/volumes" Nov 02 15:25:58.056079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:58.056047 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c"] Nov 02 15:25:58.102246 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:58.102215 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a946b8c7-ths4c"] Nov 02 15:25:59.537883 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:59.537843 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6"] Nov 02 15:25:59.538363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:59.538079 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" containerID="cri-o://3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45" gracePeriod=30 Nov 02 15:25:59.583831 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:59.583798 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm"] Nov 02 15:25:59.584049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:59.583989 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerName="istio-proxy" containerID="cri-o://b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4" gracePeriod=30 Nov 02 15:25:59.924481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:25:59.924442 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b2bb10-1a41-453e-b6f3-fecfcd0e613d" path="/var/lib/kubelet/pods/a9b2bb10-1a41-453e-b6f3-fecfcd0e613d/volumes" Nov 02 15:26:03.431110 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.431076 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc"] Nov 02 15:26:03.431523 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.431352 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" podUID="6056636e-d5c8-4b41-ac22-90cfa67a9b87" containerName="maas-api" containerID="cri-o://4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812" gracePeriod=30 Nov 02 15:26:03.745535 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.745507 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:26:03.871804 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.871766 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-metrics-certs\") pod \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " Nov 02 15:26:03.872009 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.871830 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-maas-api-tls\") pod \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " Nov 02 15:26:03.872091 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.872024 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6056636e-d5c8-4b41-ac22-90cfa67a9b87-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6056636e-d5c8-4b41-ac22-90cfa67a9b87-openshift-service-ca\") pod \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " Nov 02 15:26:03.872289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.872264 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6056636e-d5c8-4b41-ac22-90cfa67a9b87-kube-api-access-z6q5x\" (UniqueName: \"kubernetes.io/projected/6056636e-d5c8-4b41-ac22-90cfa67a9b87-kube-api-access-z6q5x\") pod \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\" (UID: \"6056636e-d5c8-4b41-ac22-90cfa67a9b87\") " Nov 02 15:26:03.872593 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.872547 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6056636e-d5c8-4b41-ac22-90cfa67a9b87-openshift-service-ca" pod "6056636e-d5c8-4b41-ac22-90cfa67a9b87" (UID: "6056636e-d5c8-4b41-ac22-90cfa67a9b87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:26:03.873316 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.873290 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6056636e-d5c8-4b41-ac22-90cfa67a9b87-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:03.874494 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.874464 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-maas-api-tls" pod "6056636e-d5c8-4b41-ac22-90cfa67a9b87" (UID: "6056636e-d5c8-4b41-ac22-90cfa67a9b87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:26:03.874591 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.874560 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-metrics-certs" pod "6056636e-d5c8-4b41-ac22-90cfa67a9b87" (UID: "6056636e-d5c8-4b41-ac22-90cfa67a9b87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:26:03.879465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.879439 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6056636e-d5c8-4b41-ac22-90cfa67a9b87-kube-api-access-z6q5x" pod "6056636e-d5c8-4b41-ac22-90cfa67a9b87" (UID: "6056636e-d5c8-4b41-ac22-90cfa67a9b87"). InnerVolumeSpecName "kube-api-access-z6q5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:03.976237 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.976158 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:03.976463 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.976440 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6056636e-d5c8-4b41-ac22-90cfa67a9b87-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:03.976571 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:03.976552 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6q5x\" (UniqueName: \"kubernetes.io/projected/6056636e-d5c8-4b41-ac22-90cfa67a9b87-kube-api-access-z6q5x\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:04.282556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:04.282466 5476 generic.go:356] "Generic (PLEG): container finished" podID="6056636e-d5c8-4b41-ac22-90cfa67a9b87" containerID="4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812" exitCode=0 Nov 02 15:26:04.282556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:04.282524 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" Nov 02 15:26:04.282556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:04.282542 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" event={"ID":"6056636e-d5c8-4b41-ac22-90cfa67a9b87","Type":"ContainerDied","Data":"4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812"} Nov 02 15:26:04.282800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:04.282635 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc" event={"ID":"6056636e-d5c8-4b41-ac22-90cfa67a9b87","Type":"ContainerDied","Data":"a131e46129fc5c55302f8a25bda4215d6ed4918aabd97e3174b33e8748508a62"} Nov 02 15:26:04.282800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:04.282665 5476 scope.go:122] "RemoveContainer" containerID="4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812" Nov 02 15:26:04.294432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:04.294412 5476 scope.go:122] "RemoveContainer" containerID="4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812" Nov 02 15:26:04.294728 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:26:04.294704 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812\": container with ID starting with 4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812 not found: ID does not exist" containerID="4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812" Nov 02 15:26:04.294780 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:04.294738 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812"} err="failed to get container status \"4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812\": rpc error: code = NotFound desc = could not find container \"4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812\": container with ID starting with 4e82e1ef5416f67ad5e3a86f2437140a92c18d897f2c960a88f51b132aafe812 not found: ID does not exist" Nov 02 15:26:04.480021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:04.479598 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc"] Nov 02 15:26:04.561251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:04.560669 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-89017f75-6d9bdc98b8-dzhxc"] Nov 02 15:26:04.868760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:04.868735 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:26:04.872163 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:04.872143 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:26:05.009238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009205 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-socket\") pod \"29ca94ef-f26d-483d-b291-11a93cc76fec\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " Nov 02 15:26:05.009536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009265 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-kube-api-access-bqjwn\" (UniqueName: \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-kube-api-access-bqjwn\") pod \"29ca94ef-f26d-483d-b291-11a93cc76fec\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " Nov 02 15:26:05.009536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009294 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ca94ef-f26d-483d-b291-11a93cc76fec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ca94ef-f26d-483d-b291-11a93cc76fec-istiod-ca-cert\") pod \"29ca94ef-f26d-483d-b291-11a93cc76fec\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " Nov 02 15:26:05.009536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009324 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-kube-api-access-wlpkt\" (UniqueName: \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-kube-api-access-wlpkt\") pod \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " Nov 02 15:26:05.009536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009384 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-token\" (UniqueName: \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-token\") pod \"29ca94ef-f26d-483d-b291-11a93cc76fec\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " Nov 02 15:26:05.009536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009415 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-data\") pod \"29ca94ef-f26d-483d-b291-11a93cc76fec\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " Nov 02 15:26:05.009536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009481 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-envoy\") pod \"29ca94ef-f26d-483d-b291-11a93cc76fec\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " Nov 02 15:26:05.009536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009513 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-data\") pod \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " Nov 02 15:26:05.009536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009526 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-socket" pod "29ca94ef-f26d-483d-b291-11a93cc76fec" (UID: "29ca94ef-f26d-483d-b291-11a93cc76fec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.010153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009544 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-podinfo\") pod \"29ca94ef-f26d-483d-b291-11a93cc76fec\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " Nov 02 15:26:05.010153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009630 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-socket\") pod \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " Nov 02 15:26:05.010153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009662 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-credential-socket\") pod \"29ca94ef-f26d-483d-b291-11a93cc76fec\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " Nov 02 15:26:05.010153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009761 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istiod-ca-cert\") pod \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " Nov 02 15:26:05.010153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009824 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-envoy\") pod \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " Nov 02 15:26:05.010153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009870 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-credential-socket\") pod \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " Nov 02 15:26:05.010153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.009926 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-certs\") pod \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " Nov 02 15:26:05.010153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.010045 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-certs\") pod \"29ca94ef-f26d-483d-b291-11a93cc76fec\" (UID: \"29ca94ef-f26d-483d-b291-11a93cc76fec\") " Nov 02 15:26:05.012116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.010063 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ca94ef-f26d-483d-b291-11a93cc76fec-istiod-ca-cert" pod "29ca94ef-f26d-483d-b291-11a93cc76fec" (UID: "29ca94ef-f26d-483d-b291-11a93cc76fec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:26:05.012116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.010105 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-token\" (UniqueName: \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-token\") pod \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " Nov 02 15:26:05.012284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.010530 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-data" pod "29ca94ef-f26d-483d-b291-11a93cc76fec" (UID: "29ca94ef-f26d-483d-b291-11a93cc76fec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.012284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.010706 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-credential-socket" pod "29ca94ef-f26d-483d-b291-11a93cc76fec" (UID: "29ca94ef-f26d-483d-b291-11a93cc76fec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.012284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.010891 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-socket" pod "c82cd674-a934-4d35-8ca1-7ba8850d1a0a" (UID: "c82cd674-a934-4d35-8ca1-7ba8850d1a0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.012284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.011097 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-credential-socket" pod "c82cd674-a934-4d35-8ca1-7ba8850d1a0a" (UID: "c82cd674-a934-4d35-8ca1-7ba8850d1a0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.012284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.011467 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istiod-ca-cert" pod "c82cd674-a934-4d35-8ca1-7ba8850d1a0a" (UID: "c82cd674-a934-4d35-8ca1-7ba8850d1a0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:26:05.012284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.011722 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-data" pod "c82cd674-a934-4d35-8ca1-7ba8850d1a0a" (UID: "c82cd674-a934-4d35-8ca1-7ba8850d1a0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.012284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.011893 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-certs" pod "c82cd674-a934-4d35-8ca1-7ba8850d1a0a" (UID: "c82cd674-a934-4d35-8ca1-7ba8850d1a0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.012284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.012092 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-certs" pod "29ca94ef-f26d-483d-b291-11a93cc76fec" (UID: "29ca94ef-f26d-483d-b291-11a93cc76fec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.012284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.012199 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-podinfo\") pod \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\" (UID: \"c82cd674-a934-4d35-8ca1-7ba8850d1a0a\") " Nov 02 15:26:05.012934 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.012907 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ca94ef-f26d-483d-b291-11a93cc76fec-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.013019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.012942 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.013019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.012956 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.013019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.012971 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.013019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.012985 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.013019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.013013 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.013277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.013026 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.013277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.013038 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.013277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.013052 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.013277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.013065 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.013277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.013169 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-token" pod "c82cd674-a934-4d35-8ca1-7ba8850d1a0a" (UID: "c82cd674-a934-4d35-8ca1-7ba8850d1a0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:05.014648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.014615 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-podinfo" pod "c82cd674-a934-4d35-8ca1-7ba8850d1a0a" (UID: "c82cd674-a934-4d35-8ca1-7ba8850d1a0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:26:05.015153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.015130 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-token" pod "29ca94ef-f26d-483d-b291-11a93cc76fec" (UID: "29ca94ef-f26d-483d-b291-11a93cc76fec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:05.015803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.015779 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-envoy" pod "c82cd674-a934-4d35-8ca1-7ba8850d1a0a" (UID: "c82cd674-a934-4d35-8ca1-7ba8850d1a0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.016732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.016712 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-podinfo" pod "29ca94ef-f26d-483d-b291-11a93cc76fec" (UID: "29ca94ef-f26d-483d-b291-11a93cc76fec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:26:05.021207 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.021183 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-envoy" pod "29ca94ef-f26d-483d-b291-11a93cc76fec" (UID: "29ca94ef-f26d-483d-b291-11a93cc76fec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:05.021280 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.021228 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-kube-api-access-bqjwn" pod "29ca94ef-f26d-483d-b291-11a93cc76fec" (UID: "29ca94ef-f26d-483d-b291-11a93cc76fec"). InnerVolumeSpecName "kube-api-access-bqjwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:05.022390 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.022372 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-kube-api-access-wlpkt" pod "c82cd674-a934-4d35-8ca1-7ba8850d1a0a" (UID: "c82cd674-a934-4d35-8ca1-7ba8850d1a0a"). InnerVolumeSpecName "kube-api-access-wlpkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:05.115227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.115194 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqjwn\" (UniqueName: \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-kube-api-access-bqjwn\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.115227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.115223 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlpkt\" (UniqueName: \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-kube-api-access-wlpkt\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.115227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.115233 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.115539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.115242 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.115539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.115257 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ca94ef-f26d-483d-b291-11a93cc76fec-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.115539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.115268 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.115539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.115276 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.115539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.115284 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c82cd674-a934-4d35-8ca1-7ba8850d1a0a-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:05.296037 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.295959 5476 generic.go:356] "Generic (PLEG): container finished" podID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerID="b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4" exitCode=0 Nov 02 15:26:05.296229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.296055 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" Nov 02 15:26:05.296229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.296146 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" event={"ID":"c82cd674-a934-4d35-8ca1-7ba8850d1a0a","Type":"ContainerDied","Data":"b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4"} Nov 02 15:26:05.296229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.296199 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm" event={"ID":"c82cd674-a934-4d35-8ca1-7ba8850d1a0a","Type":"ContainerDied","Data":"7bc3650f5fe2dd2a02491f26e4401f59af918c845b51528f3e94c87e23fbfaf3"} Nov 02 15:26:05.296422 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.296230 5476 scope.go:122] "RemoveContainer" containerID="b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4" Nov 02 15:26:05.300035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.300008 5476 generic.go:356] "Generic (PLEG): container finished" podID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerID="3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45" exitCode=0 Nov 02 15:26:05.300161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.300111 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" Nov 02 15:26:05.300231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.300210 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" event={"ID":"29ca94ef-f26d-483d-b291-11a93cc76fec","Type":"ContainerDied","Data":"3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45"} Nov 02 15:26:05.300296 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.300259 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6" event={"ID":"29ca94ef-f26d-483d-b291-11a93cc76fec","Type":"ContainerDied","Data":"6e1e73b7a281a2d19cb2dfd37f1c156a385970db9cd05481e8cf0323f2a87c6d"} Nov 02 15:26:05.314479 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.313201 5476 scope.go:122] "RemoveContainer" containerID="b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4" Nov 02 15:26:05.314479 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:26:05.314418 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4\": container with ID starting with b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4 not found: ID does not exist" containerID="b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4" Nov 02 15:26:05.314479 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.314452 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4"} err="failed to get container status \"b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4\": rpc error: code = NotFound desc = could not find container \"b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4\": container with ID starting with b4ba016a96fb87340e04a85a34ee6d28e97730f3ff70d1232e591012afd8d3a4 not found: ID does not exist" Nov 02 15:26:05.314479 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.314474 5476 scope.go:122] "RemoveContainer" containerID="3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45" Nov 02 15:26:05.325400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.325381 5476 scope.go:122] "RemoveContainer" containerID="3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45" Nov 02 15:26:05.325663 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:26:05.325646 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45\": container with ID starting with 3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45 not found: ID does not exist" containerID="3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45" Nov 02 15:26:05.325711 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.325669 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45"} err="failed to get container status \"3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45\": rpc error: code = NotFound desc = could not find container \"3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45\": container with ID starting with 3e95cbddf82dd8b032f862e549c546fd252bffb9464d87ea8d87aec22cb27d45 not found: ID does not exist" Nov 02 15:26:05.455904 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.455816 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6"] Nov 02 15:26:05.460224 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.460190 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvhn6"] Nov 02 15:26:05.561164 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.561127 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm"] Nov 02 15:26:05.569013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.568508 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-6dfkm"] Nov 02 15:26:05.951940 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.951902 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" path="/var/lib/kubelet/pods/29ca94ef-f26d-483d-b291-11a93cc76fec/volumes" Nov 02 15:26:05.952981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.952954 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6056636e-d5c8-4b41-ac22-90cfa67a9b87" path="/var/lib/kubelet/pods/6056636e-d5c8-4b41-ac22-90cfa67a9b87/volumes" Nov 02 15:26:05.953851 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:05.953830 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" path="/var/lib/kubelet/pods/c82cd674-a934-4d35-8ca1-7ba8850d1a0a/volumes" Nov 02 15:26:11.054834 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:11.054802 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb"] Nov 02 15:26:11.083660 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:11.083626 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f8f74c-h9wnb"] Nov 02 15:26:11.924967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:11.924933 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad663c1c-ec73-4f47-b9ca-55d519911296" path="/var/lib/kubelet/pods/ad663c1c-ec73-4f47-b9ca-55d519911296/volumes" Nov 02 15:26:18.070854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:18.070823 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr"] Nov 02 15:26:18.113986 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:18.113949 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ccbb111-46jkr"] Nov 02 15:26:18.458636 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:18.458606 5476 scope.go:122] "RemoveContainer" containerID="22b7605ba99a347323006c73851a3691f96dfc06f4602e2efa701584524f174c" Nov 02 15:26:18.469138 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:18.469117 5476 scope.go:122] "RemoveContainer" containerID="830e480c8eb6eaf6209e8bd386b61d984573e70bf36bbfc724a48fa5e9662bd0" Nov 02 15:26:18.478488 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:18.478468 5476 scope.go:122] "RemoveContainer" containerID="12fa8bd02d9d41ed01458d98a75fc8179ac8f7b0483d09b5d4b8915a8f4f8038" Nov 02 15:26:19.668761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.668712 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss"] Nov 02 15:26:19.669467 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669447 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6f0dccdd-426e-40a4-b92e-6ab31b55a660" containerName="payload-pre-processing" Nov 02 15:26:19.669538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669468 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0dccdd-426e-40a4-b92e-6ab31b55a660" containerName="payload-pre-processing" Nov 02 15:26:19.669538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669484 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerName="istio-proxy" Nov 02 15:26:19.669538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669489 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerName="istio-proxy" Nov 02 15:26:19.669538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669504 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6069c8d1-a8ea-480b-828c-850bfec695c8" containerName="maas-api" Nov 02 15:26:19.669538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669509 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6069c8d1-a8ea-480b-828c-850bfec695c8" containerName="maas-api" Nov 02 15:26:19.669538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669521 5476 cpu_manager.go:414] "containerMap: removing container" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" Nov 02 15:26:19.669538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669526 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" Nov 02 15:26:19.669886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669564 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6056636e-d5c8-4b41-ac22-90cfa67a9b87" containerName="maas-api" Nov 02 15:26:19.669886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669570 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6056636e-d5c8-4b41-ac22-90cfa67a9b87" containerName="maas-api" Nov 02 15:26:19.669886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669661 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0dccdd-426e-40a4-b92e-6ab31b55a660" containerName="payload-pre-processing" Nov 02 15:26:19.669886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669669 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82cd674-a934-4d35-8ca1-7ba8850d1a0a" containerName="istio-proxy" Nov 02 15:26:19.669886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669677 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6069c8d1-a8ea-480b-828c-850bfec695c8" containerName="maas-api" Nov 02 15:26:19.669886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669684 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ca94ef-f26d-483d-b291-11a93cc76fec" containerName="istio-proxy" Nov 02 15:26:19.669886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.669693 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6056636e-d5c8-4b41-ac22-90cfa67a9b87" containerName="maas-api" Nov 02 15:26:19.675490 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.675469 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.738078 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.738044 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss"] Nov 02 15:26:19.759540 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.759495 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.759957 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.759931 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.837557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.811717 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rds\" (UniqueName: \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-kube-api-access-l4rds\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.837557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.811773 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.837557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.811805 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.837557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.811912 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.837557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.811964 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.837557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.812198 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.837557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.812338 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.837557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.776344 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5aeac88e-openshift-default-dockercfg-hfkbc\"" Nov 02 15:26:19.929144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.929046 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.929322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.929166 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.929322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.929224 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.929322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.929281 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rds\" (UniqueName: \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-kube-api-access-l4rds\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.929322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.929309 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.929549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.929337 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.929549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.929421 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.929549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.929469 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.929700 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.929643 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.929823 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.929794 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.930121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.930092 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.930121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.930185 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.930569 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.930392 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.931034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.931010 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.952587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.952426 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.955257 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.955207 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.958372 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.958344 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5afb5e83-9181-464a-bf1a-81a1a42d43ee" path="/var/lib/kubelet/pods/5afb5e83-9181-464a-bf1a-81a1a42d43ee/volumes" Nov 02 15:26:19.990290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.990258 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:19.993298 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:19.993267 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rds\" (UniqueName: \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-kube-api-access-l4rds\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:20.151078 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:26:20.151037 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:26:20.291209 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:20.291113 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:20.732594 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:20.732559 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss"] Nov 02 15:26:20.771610 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:26:20.771578 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e6b2ff_a84f_428d_9df4_cb020df76d2f.slice/crio-016c58fcec885be7b4950309f9d0ce3ce702bffddf65872d8c91322c908a141d WatchSource:0}: Error finding container 016c58fcec885be7b4950309f9d0ce3ce702bffddf65872d8c91322c908a141d: Status 404 returned error can't find the container with id 016c58fcec885be7b4950309f9d0ce3ce702bffddf65872d8c91322c908a141d Nov 02 15:26:20.774206 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:20.774173 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:20.774279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:20.774239 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:20.774279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:20.774270 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:21.061211 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:21.061124 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz"] Nov 02 15:26:21.065898 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:21.065867 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2844d9c8-mtrlz"] Nov 02 15:26:21.454459 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:21.454413 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" event={"ID":"c3e6b2ff-a84f-428d-9df4-cb020df76d2f","Type":"ContainerStarted","Data":"c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd"} Nov 02 15:26:21.454613 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:21.454483 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" event={"ID":"c3e6b2ff-a84f-428d-9df4-cb020df76d2f","Type":"ContainerStarted","Data":"016c58fcec885be7b4950309f9d0ce3ce702bffddf65872d8c91322c908a141d"} Nov 02 15:26:21.570770 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:21.570719 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" podStartSLOduration=2.570705368 podStartE2EDuration="2.570705368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:21.570368082 +0000 UTC m=+1683.773246145" watchObservedRunningTime="2026-11-02 15:26:21.570705368 +0000 UTC m=+1683.773583427" Nov 02 15:26:21.924948 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:21.924910 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5f7022-ae6a-4ecb-a28e-489226637f0e" path="/var/lib/kubelet/pods/7f5f7022-ae6a-4ecb-a28e-489226637f0e/volumes" Nov 02 15:26:22.291814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:22.291734 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:22.297540 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:22.297516 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:22.462788 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:22.462753 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:22.463628 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:22.463604 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:26:24.050638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:24.050606 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh"] Nov 02 15:26:24.096509 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:24.096454 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ed2e6d3-2c5kh"] Nov 02 15:26:25.927805 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:25.927775 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b69522a-eaa1-4839-aa5e-221caa4cef34" path="/var/lib/kubelet/pods/3b69522a-eaa1-4839-aa5e-221caa4cef34/volumes" Nov 02 15:26:33.581036 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.581005 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g"] Nov 02 15:26:33.586069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.586047 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.623383 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.623353 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-credential-socket\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.623556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.623501 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7d2cc8-ea14-45c9-b096-578619c36bf0-istiod-ca-cert\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.623625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.623608 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-socket\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.623690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.623625 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-podinfo\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.623748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.623715 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-envoy\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.623748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.623742 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-token\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.623879 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.623866 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-data\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.623938 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.623920 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfv2z\" (UniqueName: \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-kube-api-access-rfv2z\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.624033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.623972 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-certs\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.685077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.684934 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g"] Nov 02 15:26:33.726120 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.726084 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7d2cc8-ea14-45c9-b096-578619c36bf0-istiod-ca-cert\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.727326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.727302 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-socket\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.727822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.727803 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-podinfo\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.728431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.728409 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-envoy\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.728901 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.728883 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-token\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.729444 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.729420 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-data\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.729671 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.729652 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfv2z\" (UniqueName: \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-kube-api-access-rfv2z\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.729807 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.729792 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-certs\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.730252 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.730233 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-credential-socket\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.730824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.729724 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-data\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.730961 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.727141 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7d2cc8-ea14-45c9-b096-578619c36bf0-istiod-ca-cert\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.731100 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.730161 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-certs\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.731212 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.729439 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-7ad552-openshift-default-dockercfg-rs9fn\"" Nov 02 15:26:33.731321 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.730631 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-credential-socket\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.731463 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.727755 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-socket\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.732341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.732321 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-podinfo\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.733893 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.733870 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-envoy\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.789882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.789854 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-token\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.790096 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.790074 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfv2z\" (UniqueName: \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-kube-api-access-rfv2z\") pod \"e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:33.901313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:33.901273 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:34.393092 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:34.393059 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g"] Nov 02 15:26:34.396473 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:26:34.396446 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7d2cc8_ea14_45c9_b096_578619c36bf0.slice/crio-a63f4c5bcb746a3c06397cb01cc74f2bd5688d37677149f5634b69d913a9abb6 WatchSource:0}: Error finding container a63f4c5bcb746a3c06397cb01cc74f2bd5688d37677149f5634b69d913a9abb6: Status 404 returned error can't find the container with id a63f4c5bcb746a3c06397cb01cc74f2bd5688d37677149f5634b69d913a9abb6 Nov 02 15:26:34.399064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:34.399030 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:34.399177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:34.399110 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:34.399177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:34.399154 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:34.574463 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:34.574424 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" event={"ID":"be7d2cc8-ea14-45c9-b096-578619c36bf0","Type":"ContainerStarted","Data":"a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462"} Nov 02 15:26:34.574643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:34.574492 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" event={"ID":"be7d2cc8-ea14-45c9-b096-578619c36bf0","Type":"ContainerStarted","Data":"a63f4c5bcb746a3c06397cb01cc74f2bd5688d37677149f5634b69d913a9abb6"} Nov 02 15:26:34.701066 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:34.693728 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" podStartSLOduration=1.693708332 podStartE2EDuration="1.693708332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:34.691426567 +0000 UTC m=+1696.894304628" watchObservedRunningTime="2026-11-02 15:26:34.693708332 +0000 UTC m=+1696.896586395" Nov 02 15:26:34.902331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:34.902287 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:35.081435 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:35.081357 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:26:35.907511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:35.907481 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:36.595466 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:36.595428 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:36.596430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:36.596411 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:26:45.054959 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:45.054924 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz"] Nov 02 15:26:45.101173 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:45.101140 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee0de28-d8mmz"] Nov 02 15:26:45.924494 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:45.924461 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29c38e8-501b-4c00-8a8a-01f8e92fcc67" path="/var/lib/kubelet/pods/b29c38e8-501b-4c00-8a8a-01f8e92fcc67/volumes" Nov 02 15:26:50.147490 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:26:50.147446 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:26:55.049240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:55.049210 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45"] Nov 02 15:26:55.095856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:55.095825 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw"] Nov 02 15:26:55.099891 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:55.099872 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d70a7ef0-ttt45"] Nov 02 15:26:55.103062 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:55.103040 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd3ebca-lhmhw"] Nov 02 15:26:55.925683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:55.925653 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a538d791-3c71-47d0-82aa-f11762f4470f" path="/var/lib/kubelet/pods/a538d791-3c71-47d0-82aa-f11762f4470f/volumes" Nov 02 15:26:55.926150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:55.926135 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de02b6ea-99c6-4047-827a-909cc2d60367" path="/var/lib/kubelet/pods/de02b6ea-99c6-4047-827a-909cc2d60367/volumes" Nov 02 15:26:58.061907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:58.061837 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds"] Nov 02 15:26:58.107449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:58.107420 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83a3df04-drjds"] Nov 02 15:26:59.925436 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:26:59.925398 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc86d00f-2713-4d37-993d-63e87d23f495" path="/var/lib/kubelet/pods/dc86d00f-2713-4d37-993d-63e87d23f495/volumes" Nov 02 15:27:01.053668 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:01.053635 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np"] Nov 02 15:27:01.099625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:01.099594 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f7c3da-rs9np"] Nov 02 15:27:01.925856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:01.925822 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf52a02e-c0fa-4351-89a9-7027069f4842" path="/var/lib/kubelet/pods/cf52a02e-c0fa-4351-89a9-7027069f4842/volumes" Nov 02 15:27:09.545985 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.545948 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f"] Nov 02 15:27:09.550226 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.550210 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:09.599431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.599401 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:09.599568 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.599467 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stwn7\" (UniqueName: \"kubernetes.io/projected/8bd46e27-c109-486b-a58d-eb9e058f2a59-kube-api-access-stwn7\") pod \"maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:09.599568 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.599551 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:09.646872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.646841 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:27:09.647050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.646928 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:27:09.653472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.653438 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f"] Nov 02 15:27:09.700731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.700691 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stwn7\" (UniqueName: \"kubernetes.io/projected/8bd46e27-c109-486b-a58d-eb9e058f2a59-kube-api-access-stwn7\") pod \"maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:09.700870 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.700741 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:09.701078 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.701058 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:09.701679 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.701661 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:09.703141 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.703118 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:09.758802 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.758774 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stwn7\" (UniqueName: \"kubernetes.io/projected/8bd46e27-c109-486b-a58d-eb9e058f2a59-kube-api-access-stwn7\") pod \"maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:09.864247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:09.864226 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:10.053959 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:10.053929 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z"] Nov 02 15:27:10.091483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:10.091452 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f23e94e-zzv4z"] Nov 02 15:27:10.243049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:10.242967 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f"] Nov 02 15:27:10.247573 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:27:10.247546 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd46e27_c109_486b_a58d_eb9e058f2a59.slice/crio-ae772605b7e66d476a1bda5676beaa3c4df066ca4d9d5715b6aff8cc88bf2d5c WatchSource:0}: Error finding container ae772605b7e66d476a1bda5676beaa3c4df066ca4d9d5715b6aff8cc88bf2d5c: Status 404 returned error can't find the container with id ae772605b7e66d476a1bda5676beaa3c4df066ca4d9d5715b6aff8cc88bf2d5c Nov 02 15:27:10.892775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:10.892734 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" event={"ID":"8bd46e27-c109-486b-a58d-eb9e058f2a59","Type":"ContainerStarted","Data":"d6d3d059764d126c0a92a0b3454914bbdbe9d851ce758c5a1f76865b1b5ea5be"} Nov 02 15:27:10.893197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:10.892786 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" event={"ID":"8bd46e27-c109-486b-a58d-eb9e058f2a59","Type":"ContainerStarted","Data":"ae772605b7e66d476a1bda5676beaa3c4df066ca4d9d5715b6aff8cc88bf2d5c"} Nov 02 15:27:11.005300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:11.005192 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" podStartSLOduration=2.005178789 podStartE2EDuration="2.005178789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:11.001395256 +0000 UTC m=+1733.204273316" watchObservedRunningTime="2026-11-02 15:27:11.005178789 +0000 UTC m=+1733.208056849" Nov 02 15:27:11.902362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:11.902328 5476 generic.go:356] "Generic (PLEG): container finished" podID="8bd46e27-c109-486b-a58d-eb9e058f2a59" containerID="d6d3d059764d126c0a92a0b3454914bbdbe9d851ce758c5a1f76865b1b5ea5be" exitCode=0 Nov 02 15:27:11.902744 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:11.902391 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" event={"ID":"8bd46e27-c109-486b-a58d-eb9e058f2a59","Type":"ContainerDied","Data":"d6d3d059764d126c0a92a0b3454914bbdbe9d851ce758c5a1f76865b1b5ea5be"} Nov 02 15:27:11.925089 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:11.925061 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed99465-e804-4ae5-8237-67888b99672e" path="/var/lib/kubelet/pods/7ed99465-e804-4ae5-8237-67888b99672e/volumes" Nov 02 15:27:13.086557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.086532 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:13.147421 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.147398 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-openshift-ca\") pod \"8bd46e27-c109-486b-a58d-eb9e058f2a59\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " Nov 02 15:27:13.147620 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.147599 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-xks-ca\") pod \"8bd46e27-c109-486b-a58d-eb9e058f2a59\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " Nov 02 15:27:13.147730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.147701 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-openshift-ca" pod "8bd46e27-c109-486b-a58d-eb9e058f2a59" (UID: "8bd46e27-c109-486b-a58d-eb9e058f2a59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:13.147866 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.147716 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd46e27-c109-486b-a58d-eb9e058f2a59-kube-api-access-stwn7\" (UniqueName: \"kubernetes.io/projected/8bd46e27-c109-486b-a58d-eb9e058f2a59-kube-api-access-stwn7\") pod \"8bd46e27-c109-486b-a58d-eb9e058f2a59\" (UID: \"8bd46e27-c109-486b-a58d-eb9e058f2a59\") " Nov 02 15:27:13.148396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.148375 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:13.149874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.149837 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd46e27-c109-486b-a58d-eb9e058f2a59-kube-api-access-stwn7" pod "8bd46e27-c109-486b-a58d-eb9e058f2a59" (UID: "8bd46e27-c109-486b-a58d-eb9e058f2a59"). InnerVolumeSpecName "kube-api-access-stwn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:13.149983 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.149941 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-xks-ca" pod "8bd46e27-c109-486b-a58d-eb9e058f2a59" (UID: "8bd46e27-c109-486b-a58d-eb9e058f2a59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:27:13.249728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.249654 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd46e27-c109-486b-a58d-eb9e058f2a59-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:13.249728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.249679 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stwn7\" (UniqueName: \"kubernetes.io/projected/8bd46e27-c109-486b-a58d-eb9e058f2a59-kube-api-access-stwn7\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:13.922788 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.922759 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" Nov 02 15:27:13.925604 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.925576 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f" event={"ID":"8bd46e27-c109-486b-a58d-eb9e058f2a59","Type":"ContainerDied","Data":"ae772605b7e66d476a1bda5676beaa3c4df066ca4d9d5715b6aff8cc88bf2d5c"} Nov 02 15:27:13.925735 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:13.925617 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae772605b7e66d476a1bda5676beaa3c4df066ca4d9d5715b6aff8cc88bf2d5c" Nov 02 15:27:17.000843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:17.000810 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl"] Nov 02 15:27:17.001258 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:17.001058 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" containerID="cri-o://dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2" gracePeriod=30 Nov 02 15:27:18.544615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:18.544585 5476 scope.go:122] "RemoveContainer" containerID="ceeac6a8847a1cc1c28c7efca3d148ac63e57feb32b51f2a8b43fece632d9725" Nov 02 15:27:18.555509 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:18.555482 5476 scope.go:122] "RemoveContainer" containerID="f561494bd9666258551ae6a8b25fc446a52742123f3693cf34c235e6c42dcb26" Nov 02 15:27:18.564759 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:18.564736 5476 scope.go:122] "RemoveContainer" containerID="d5387157f43eba1d666743eb7fb98183358f19c9e3856d8b6527916e63a87703" Nov 02 15:27:18.575032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:18.574984 5476 scope.go:122] "RemoveContainer" containerID="ae95e83d20ec95ed11f41b6d797f8c2d0b47ea4c24524fbbeaabfd45e2a4c073" Nov 02 15:27:18.587233 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:18.587096 5476 scope.go:122] "RemoveContainer" containerID="5d5f72ab477cb931cd34801c8194411c56c10b52a4730b0807250ca8e5705342" Nov 02 15:27:18.597455 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:18.597428 5476 scope.go:122] "RemoveContainer" containerID="573f2e023ba8f4cca2d7105a66c4a48838157c0083fb9819432277e2a3a7280c" Nov 02 15:27:18.606205 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:18.606190 5476 scope.go:122] "RemoveContainer" containerID="323c8ed34701b24465a6e295079c8b36d7344461c3cb0379536a1e4fbe91474a" Nov 02 15:27:19.927212 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:19.927183 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd"] Nov 02 15:27:19.927838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:19.927823 5476 cpu_manager.go:414] "containerMap: removing container" podUID="8bd46e27-c109-486b-a58d-eb9e058f2a59" containerName="revoke-keys" Nov 02 15:27:19.927926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:19.927839 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd46e27-c109-486b-a58d-eb9e058f2a59" containerName="revoke-keys" Nov 02 15:27:19.927963 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:19.927954 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd46e27-c109-486b-a58d-eb9e058f2a59" containerName="revoke-keys" Nov 02 15:27:19.933058 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:19.933039 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:19.982805 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:19.982781 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:27:19.985821 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:19.985804 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:27:20.036174 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.036147 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd"] Nov 02 15:27:20.064472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.064447 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29443935-t6vdd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:20.064665 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.064647 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqsbb\" (UniqueName: \"kubernetes.io/projected/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-kube-api-access-nqsbb\") pod \"maas-api-revoke-sub-29443935-t6vdd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:20.064807 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.064788 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29443935-t6vdd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:20.148013 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:27:20.147972 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:27:20.166023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.165967 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29443935-t6vdd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:20.166175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.166140 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29443935-t6vdd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:20.166313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.166295 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqsbb\" (UniqueName: \"kubernetes.io/projected/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-kube-api-access-nqsbb\") pod \"maas-api-revoke-sub-29443935-t6vdd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:20.166715 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.166686 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29443935-t6vdd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:20.168631 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.168603 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29443935-t6vdd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:20.224218 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.224154 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqsbb\" (UniqueName: \"kubernetes.io/projected/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-kube-api-access-nqsbb\") pod \"maas-api-revoke-sub-29443935-t6vdd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:20.247435 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.247411 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:20.634197 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:27:20.634157 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccea56fd_e01e_4948_b2a5_1bc7d0c263bd.slice/crio-f496c4da3a0a08a270ee9e25cfef77a992530f2f4adc007348d3f0572c6b0a82 WatchSource:0}: Error finding container f496c4da3a0a08a270ee9e25cfef77a992530f2f4adc007348d3f0572c6b0a82: Status 404 returned error can't find the container with id f496c4da3a0a08a270ee9e25cfef77a992530f2f4adc007348d3f0572c6b0a82 Nov 02 15:27:20.634614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.634566 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd"] Nov 02 15:27:20.978006 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:20.977959 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" event={"ID":"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd","Type":"ContainerStarted","Data":"f496c4da3a0a08a270ee9e25cfef77a992530f2f4adc007348d3f0572c6b0a82"} Nov 02 15:27:21.987570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:21.987529 5476 generic.go:356] "Generic (PLEG): container finished" podID="ccea56fd-e01e-4948-b2a5-1bc7d0c263bd" containerID="a1e2792ae7344fd6c30404f62609b06352ad43eb7496212371cb55954f2feddb" exitCode=0 Nov 02 15:27:21.987969 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:21.987594 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" event={"ID":"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd","Type":"ContainerDied","Data":"a1e2792ae7344fd6c30404f62609b06352ad43eb7496212371cb55954f2feddb"} Nov 02 15:27:22.304938 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.304911 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:27:22.501689 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.501650 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-socket\") pod \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " Nov 02 15:27:22.501866 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.501738 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-credential-socket\") pod \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " Nov 02 15:27:22.501866 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.501792 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istiod-ca-cert\") pod \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " Nov 02 15:27:22.502028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.501889 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-data\") pod \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " Nov 02 15:27:22.502028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.501983 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-socket" pod "a6c27ff3-82e0-4411-ab70-c2189b9cf795" (UID: "a6c27ff3-82e0-4411-ab70-c2189b9cf795"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.502156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.502058 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-kube-api-access-dpszx\" (UniqueName: \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-kube-api-access-dpszx\") pod \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " Nov 02 15:27:22.502156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.502108 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-credential-socket" pod "a6c27ff3-82e0-4411-ab70-c2189b9cf795" (UID: "a6c27ff3-82e0-4411-ab70-c2189b9cf795"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.502458 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.502345 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-certs\") pod \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " Nov 02 15:27:22.502458 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.502454 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-podinfo\") pod \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " Nov 02 15:27:22.502630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.502501 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-envoy\") pod \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " Nov 02 15:27:22.502926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.502897 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-data" pod "a6c27ff3-82e0-4411-ab70-c2189b9cf795" (UID: "a6c27ff3-82e0-4411-ab70-c2189b9cf795"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.503087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.502902 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-token\" (UniqueName: \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-token\") pod \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\" (UID: \"a6c27ff3-82e0-4411-ab70-c2189b9cf795\") " Nov 02 15:27:22.504070 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.504043 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.504184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.504075 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.504184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.504090 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.504581 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.504557 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istiod-ca-cert" pod "a6c27ff3-82e0-4411-ab70-c2189b9cf795" (UID: "a6c27ff3-82e0-4411-ab70-c2189b9cf795"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:22.504893 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.504835 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-kube-api-access-dpszx" pod "a6c27ff3-82e0-4411-ab70-c2189b9cf795" (UID: "a6c27ff3-82e0-4411-ab70-c2189b9cf795"). InnerVolumeSpecName "kube-api-access-dpszx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:22.505454 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.505328 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-certs" pod "a6c27ff3-82e0-4411-ab70-c2189b9cf795" (UID: "a6c27ff3-82e0-4411-ab70-c2189b9cf795"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.505816 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.505789 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-token" pod "a6c27ff3-82e0-4411-ab70-c2189b9cf795" (UID: "a6c27ff3-82e0-4411-ab70-c2189b9cf795"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:22.506976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.506944 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-envoy" pod "a6c27ff3-82e0-4411-ab70-c2189b9cf795" (UID: "a6c27ff3-82e0-4411-ab70-c2189b9cf795"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:22.507125 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.507087 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-podinfo" pod "a6c27ff3-82e0-4411-ab70-c2189b9cf795" (UID: "a6c27ff3-82e0-4411-ab70-c2189b9cf795"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:22.605886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.605852 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.605886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.605880 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.605886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.605889 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.606147 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.605898 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.606147 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.605906 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6c27ff3-82e0-4411-ab70-c2189b9cf795-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:22.606147 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:22.605915 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpszx\" (UniqueName: \"kubernetes.io/projected/a6c27ff3-82e0-4411-ab70-c2189b9cf795-kube-api-access-dpszx\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:23.000823 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.000790 5476 generic.go:356] "Generic (PLEG): container finished" podID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerID="dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2" exitCode=0 Nov 02 15:27:23.001279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.000851 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" Nov 02 15:27:23.001279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.000874 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" event={"ID":"a6c27ff3-82e0-4411-ab70-c2189b9cf795","Type":"ContainerDied","Data":"dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2"} Nov 02 15:27:23.001279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.000943 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl" event={"ID":"a6c27ff3-82e0-4411-ab70-c2189b9cf795","Type":"ContainerDied","Data":"e6f7553f19b740dc0e79305cb67ce83d3df9a0b6a9412ec9a22ff129aa52899c"} Nov 02 15:27:23.001279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.000972 5476 scope.go:122] "RemoveContainer" containerID="dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2" Nov 02 15:27:23.015304 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.015281 5476 scope.go:122] "RemoveContainer" containerID="dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2" Nov 02 15:27:23.015625 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:27:23.015598 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2\": container with ID starting with dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2 not found: ID does not exist" containerID="dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2" Nov 02 15:27:23.015695 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.015639 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2"} err="failed to get container status \"dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2\": rpc error: code = NotFound desc = could not find container \"dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2\": container with ID starting with dc0029dee3d47ed1e3d18ff936fb09d2000592d4cb05b4a1f50129a4da9b30d2 not found: ID does not exist" Nov 02 15:27:23.159514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.159484 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl"] Nov 02 15:27:23.171707 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.171682 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:23.205301 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.205276 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-m5vsl"] Nov 02 15:27:23.216063 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.216031 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-xks-ca\") pod \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " Nov 02 15:27:23.216263 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.216244 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-kube-api-access-nqsbb\" (UniqueName: \"kubernetes.io/projected/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-kube-api-access-nqsbb\") pod \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " Nov 02 15:27:23.216372 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.216356 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-openshift-ca\") pod \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\" (UID: \"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd\") " Nov 02 15:27:23.216758 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.216736 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-openshift-ca" pod "ccea56fd-e01e-4948-b2a5-1bc7d0c263bd" (UID: "ccea56fd-e01e-4948-b2a5-1bc7d0c263bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:23.217059 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.217043 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:23.218133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.218109 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-xks-ca" pod "ccea56fd-e01e-4948-b2a5-1bc7d0c263bd" (UID: "ccea56fd-e01e-4948-b2a5-1bc7d0c263bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:27:23.218261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.218246 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-kube-api-access-nqsbb" pod "ccea56fd-e01e-4948-b2a5-1bc7d0c263bd" (UID: "ccea56fd-e01e-4948-b2a5-1bc7d0c263bd"). InnerVolumeSpecName "kube-api-access-nqsbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:23.318972 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.318923 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:23.318972 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.318964 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqsbb\" (UniqueName: \"kubernetes.io/projected/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd-kube-api-access-nqsbb\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:23.925292 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:23.925251 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" path="/var/lib/kubelet/pods/a6c27ff3-82e0-4411-ab70-c2189b9cf795/volumes" Nov 02 15:27:24.011677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:24.011647 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" Nov 02 15:27:24.012166 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:24.011676 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd" event={"ID":"ccea56fd-e01e-4948-b2a5-1bc7d0c263bd","Type":"ContainerDied","Data":"f496c4da3a0a08a270ee9e25cfef77a992530f2f4adc007348d3f0572c6b0a82"} Nov 02 15:27:24.012166 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:24.011723 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f496c4da3a0a08a270ee9e25cfef77a992530f2f4adc007348d3f0572c6b0a82" Nov 02 15:27:35.640872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:35.640833 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:27:44.546118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:44.545990 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx"] Nov 02 15:27:44.569899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:44.546218 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" containerID="cri-o://95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e" gracePeriod=30 Nov 02 15:27:44.592320 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:44.592288 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f"] Nov 02 15:27:44.592564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:44.592541 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" containerID="cri-o://3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc" gracePeriod=30 Nov 02 15:27:44.596465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:44.596440 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw"] Nov 02 15:27:44.596653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:44.596619 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" containerID="cri-o://df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9" gracePeriod=30 Nov 02 15:27:44.638989 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:44.638954 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp"] Nov 02 15:27:44.639213 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:44.639188 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" containerID="cri-o://8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4" gracePeriod=30 Nov 02 15:27:48.020886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:48.020851 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:27:48.021280 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:48.020907 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:27:49.924154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:49.923808 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:27:50.041757 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.041728 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:27:50.067156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067119 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-data\") pod \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " Nov 02 15:27:50.067362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067185 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istiod-ca-cert\") pod \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " Nov 02 15:27:50.067362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067234 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8458c4f5-ccc7-4444-b053-61dd0820c344-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8458c4f5-ccc7-4444-b053-61dd0820c344-istiod-ca-cert\") pod \"8458c4f5-ccc7-4444-b053-61dd0820c344\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " Nov 02 15:27:50.067362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067268 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-podinfo\") pod \"8458c4f5-ccc7-4444-b053-61dd0820c344\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " Nov 02 15:27:50.067362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067322 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-credential-socket\") pod \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " Nov 02 15:27:50.067625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067376 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-podinfo\") pod \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " Nov 02 15:27:50.067625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067442 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-envoy\") pod \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " Nov 02 15:27:50.067625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067471 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-envoy\") pod \"8458c4f5-ccc7-4444-b053-61dd0820c344\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " Nov 02 15:27:50.067625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067517 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-kube-api-access-dkhj4\" (UniqueName: \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-kube-api-access-dkhj4\") pod \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " Nov 02 15:27:50.067625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067590 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-socket\") pod \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " Nov 02 15:27:50.067976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067634 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-certs\") pod \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " Nov 02 15:27:50.067976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067684 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-token\" (UniqueName: \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-token\") pod \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\" (UID: \"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9\") " Nov 02 15:27:50.067976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067738 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-credential-socket\") pod \"8458c4f5-ccc7-4444-b053-61dd0820c344\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " Nov 02 15:27:50.067976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067785 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-kube-api-access-rlhtx\" (UniqueName: \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-kube-api-access-rlhtx\") pod \"8458c4f5-ccc7-4444-b053-61dd0820c344\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " Nov 02 15:27:50.067976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067852 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-token\" (UniqueName: \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-token\") pod \"8458c4f5-ccc7-4444-b053-61dd0820c344\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " Nov 02 15:27:50.067976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067899 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-data\") pod \"8458c4f5-ccc7-4444-b053-61dd0820c344\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " Nov 02 15:27:50.067976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.067948 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-certs\") pod \"8458c4f5-ccc7-4444-b053-61dd0820c344\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " Nov 02 15:27:50.068473 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.068024 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-socket\") pod \"8458c4f5-ccc7-4444-b053-61dd0820c344\" (UID: \"8458c4f5-ccc7-4444-b053-61dd0820c344\") " Nov 02 15:27:50.069378 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.069347 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-socket" pod "8458c4f5-ccc7-4444-b053-61dd0820c344" (UID: "8458c4f5-ccc7-4444-b053-61dd0820c344"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.069580 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.069559 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-credential-socket" pod "8458c4f5-ccc7-4444-b053-61dd0820c344" (UID: "8458c4f5-ccc7-4444-b053-61dd0820c344"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.070239 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.070203 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-envoy" pod "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" (UID: "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.070437 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.070418 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-data" pod "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" (UID: "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.070816 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.070796 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istiod-ca-cert" pod "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" (UID: "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:50.072431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.071261 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8458c4f5-ccc7-4444-b053-61dd0820c344-istiod-ca-cert" pod "8458c4f5-ccc7-4444-b053-61dd0820c344" (UID: "8458c4f5-ccc7-4444-b053-61dd0820c344"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:50.072431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.071736 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-certs" pod "8458c4f5-ccc7-4444-b053-61dd0820c344" (UID: "8458c4f5-ccc7-4444-b053-61dd0820c344"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.072431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.071788 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-socket" pod "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" (UID: "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.072431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.071823 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-kube-api-access-rlhtx" pod "8458c4f5-ccc7-4444-b053-61dd0820c344" (UID: "8458c4f5-ccc7-4444-b053-61dd0820c344"). InnerVolumeSpecName "kube-api-access-rlhtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.072431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.071976 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-data" pod "8458c4f5-ccc7-4444-b053-61dd0820c344" (UID: "8458c4f5-ccc7-4444-b053-61dd0820c344"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.072431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.072050 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-certs" pod "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" (UID: "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.072431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.072287 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-credential-socket" pod "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" (UID: "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.072431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.072400 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-token" pod "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" (UID: "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.077753 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.077713 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-podinfo" pod "8458c4f5-ccc7-4444-b053-61dd0820c344" (UID: "8458c4f5-ccc7-4444-b053-61dd0820c344"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:50.077861 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.077810 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-token" pod "8458c4f5-ccc7-4444-b053-61dd0820c344" (UID: "8458c4f5-ccc7-4444-b053-61dd0820c344"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.080172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.080146 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-envoy" pod "8458c4f5-ccc7-4444-b053-61dd0820c344" (UID: "8458c4f5-ccc7-4444-b053-61dd0820c344"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.084726 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.084699 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-kube-api-access-dkhj4" pod "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" (UID: "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9"). InnerVolumeSpecName "kube-api-access-dkhj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.086112 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.086090 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-podinfo" pod "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" (UID: "60c43ec4-5cda-4a28-93a2-b8c12d45c0e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:50.087367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.087343 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:27:50.092818 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.092795 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:27:50.158735 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:27:50.158640 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:27:50.172513 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.172460 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istiod-ca-cert\") pod \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " Nov 02 15:27:50.172513 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.172514 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-kube-api-access-slpsl\" (UniqueName: \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-kube-api-access-slpsl\") pod \"0f36f5a9-b45a-4c86-986e-3a494915914b\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " Nov 02 15:27:50.172749 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.172553 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-socket\") pod \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " Nov 02 15:27:50.172749 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.172606 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-certs\") pod \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " Nov 02 15:27:50.172749 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.172700 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-podinfo\") pod \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " Nov 02 15:27:50.172749 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.172738 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f36f5a9-b45a-4c86-986e-3a494915914b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f36f5a9-b45a-4c86-986e-3a494915914b-istiod-ca-cert\") pod \"0f36f5a9-b45a-4c86-986e-3a494915914b\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " Nov 02 15:27:50.172909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.172776 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-envoy\") pod \"0f36f5a9-b45a-4c86-986e-3a494915914b\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " Nov 02 15:27:50.172909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.172868 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-data\") pod \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " Nov 02 15:27:50.172909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.172896 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-certs\") pod \"0f36f5a9-b45a-4c86-986e-3a494915914b\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " Nov 02 15:27:50.173107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.172939 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-token\" (UniqueName: \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-token\") pod \"0f36f5a9-b45a-4c86-986e-3a494915914b\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " Nov 02 15:27:50.173166 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.172985 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-kube-api-access-vx4lp\" (UniqueName: \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-kube-api-access-vx4lp\") pod \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " Nov 02 15:27:50.173217 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.173183 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-token\" (UniqueName: \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-token\") pod \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " Nov 02 15:27:50.173310 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.173293 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-envoy\") pod \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " Nov 02 15:27:50.173361 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.173342 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-data\") pod \"0f36f5a9-b45a-4c86-986e-3a494915914b\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " Nov 02 15:27:50.173643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.173620 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-data" pod "0f36f5a9-b45a-4c86-986e-3a494915914b" (UID: "0f36f5a9-b45a-4c86-986e-3a494915914b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.174770 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.174244 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istiod-ca-cert" pod "bcde6ebe-5ae4-413f-8cfc-d09877e3e297" (UID: "bcde6ebe-5ae4-413f-8cfc-d09877e3e297"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:50.174770 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.174562 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-certs" pod "bcde6ebe-5ae4-413f-8cfc-d09877e3e297" (UID: "bcde6ebe-5ae4-413f-8cfc-d09877e3e297"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.175175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.174804 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-socket" pod "bcde6ebe-5ae4-413f-8cfc-d09877e3e297" (UID: "bcde6ebe-5ae4-413f-8cfc-d09877e3e297"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.175175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.175142 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-data" pod "bcde6ebe-5ae4-413f-8cfc-d09877e3e297" (UID: "bcde6ebe-5ae4-413f-8cfc-d09877e3e297"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.175570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.175525 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f36f5a9-b45a-4c86-986e-3a494915914b-istiod-ca-cert" pod "0f36f5a9-b45a-4c86-986e-3a494915914b" (UID: "0f36f5a9-b45a-4c86-986e-3a494915914b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:50.175655 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.175576 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-credential-socket\") pod \"0f36f5a9-b45a-4c86-986e-3a494915914b\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " Nov 02 15:27:50.175743 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.175684 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-credential-socket\") pod \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\" (UID: \"bcde6ebe-5ae4-413f-8cfc-d09877e3e297\") " Nov 02 15:27:50.175808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.175758 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-socket\") pod \"0f36f5a9-b45a-4c86-986e-3a494915914b\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " Nov 02 15:27:50.175868 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.175815 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-podinfo\") pod \"0f36f5a9-b45a-4c86-986e-3a494915914b\" (UID: \"0f36f5a9-b45a-4c86-986e-3a494915914b\") " Nov 02 15:27:50.176563 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.176540 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-certs" pod "0f36f5a9-b45a-4c86-986e-3a494915914b" (UID: "0f36f5a9-b45a-4c86-986e-3a494915914b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.176903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.176764 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.176903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.176780 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-credential-socket" pod "bcde6ebe-5ae4-413f-8cfc-d09877e3e297" (UID: "bcde6ebe-5ae4-413f-8cfc-d09877e3e297"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.177078 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.176952 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-credential-socket" pod "0f36f5a9-b45a-4c86-986e-3a494915914b" (UID: "0f36f5a9-b45a-4c86-986e-3a494915914b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.176792 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkhj4\" (UniqueName: \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-kube-api-access-dkhj4\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177151 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-socket" pod "0f36f5a9-b45a-4c86-986e-3a494915914b" (UID: "0f36f5a9-b45a-4c86-986e-3a494915914b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177151 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-kube-api-access-vx4lp" pod "bcde6ebe-5ae4-413f-8cfc-d09877e3e297" (UID: "bcde6ebe-5ae4-413f-8cfc-d09877e3e297"). InnerVolumeSpecName "kube-api-access-vx4lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177175 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177195 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177211 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177227 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177239 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177254 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177268 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177283 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlhtx\" (UniqueName: \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-kube-api-access-rlhtx\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177297 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177310 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f36f5a9-b45a-4c86-986e-3a494915914b-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177324 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177339 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177352 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177368 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177384 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8458c4f5-ccc7-4444-b053-61dd0820c344-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177399 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177412 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177427 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8458c4f5-ccc7-4444-b053-61dd0820c344-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177443 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8458c4f5-ccc7-4444-b053-61dd0820c344-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177457 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177470 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177483 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.177519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.177496 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.179505 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.179479 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-podinfo" pod "0f36f5a9-b45a-4c86-986e-3a494915914b" (UID: "0f36f5a9-b45a-4c86-986e-3a494915914b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:50.181414 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.181390 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-kube-api-access-slpsl" pod "0f36f5a9-b45a-4c86-986e-3a494915914b" (UID: "0f36f5a9-b45a-4c86-986e-3a494915914b"). InnerVolumeSpecName "kube-api-access-slpsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.183384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.183358 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-token" pod "bcde6ebe-5ae4-413f-8cfc-d09877e3e297" (UID: "bcde6ebe-5ae4-413f-8cfc-d09877e3e297"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.183512 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.183493 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-podinfo" pod "bcde6ebe-5ae4-413f-8cfc-d09877e3e297" (UID: "bcde6ebe-5ae4-413f-8cfc-d09877e3e297"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:50.183709 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.183640 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-token" pod "0f36f5a9-b45a-4c86-986e-3a494915914b" (UID: "0f36f5a9-b45a-4c86-986e-3a494915914b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.185028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.184959 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-envoy" pod "bcde6ebe-5ae4-413f-8cfc-d09877e3e297" (UID: "bcde6ebe-5ae4-413f-8cfc-d09877e3e297"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.185247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.185226 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-envoy" pod "0f36f5a9-b45a-4c86-986e-3a494915914b" (UID: "0f36f5a9-b45a-4c86-986e-3a494915914b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.226471 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.226434 5476 generic.go:356] "Generic (PLEG): container finished" podID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerID="95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e" exitCode=0 Nov 02 15:27:50.226638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.226518 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" event={"ID":"8458c4f5-ccc7-4444-b053-61dd0820c344","Type":"ContainerDied","Data":"95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e"} Nov 02 15:27:50.226638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.226566 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" event={"ID":"8458c4f5-ccc7-4444-b053-61dd0820c344","Type":"ContainerDied","Data":"c19ac992295057eb6f615fd861dfbbc3db7e522cc907ecbd98fc96c52ca62c3d"} Nov 02 15:27:50.226638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.226594 5476 scope.go:122] "RemoveContainer" containerID="95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e" Nov 02 15:27:50.226886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.226748 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx" Nov 02 15:27:50.239159 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.239101 5476 generic.go:356] "Generic (PLEG): container finished" podID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerID="3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc" exitCode=0 Nov 02 15:27:50.239464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.239438 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" event={"ID":"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9","Type":"ContainerDied","Data":"3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc"} Nov 02 15:27:50.239542 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.239501 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" event={"ID":"60c43ec4-5cda-4a28-93a2-b8c12d45c0e9","Type":"ContainerDied","Data":"18d23779cbda51a9141cba90f5ba6754205ce21127b4e4e64a4d2b9e177a8afe"} Nov 02 15:27:50.239622 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.239596 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f" Nov 02 15:27:50.243418 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.243386 5476 generic.go:356] "Generic (PLEG): container finished" podID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerID="df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9" exitCode=0 Nov 02 15:27:50.243586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.243566 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" event={"ID":"bcde6ebe-5ae4-413f-8cfc-d09877e3e297","Type":"ContainerDied","Data":"df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9"} Nov 02 15:27:50.243658 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.243630 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" event={"ID":"bcde6ebe-5ae4-413f-8cfc-d09877e3e297","Type":"ContainerDied","Data":"6dd8e4ab08d98325f9d145856efab215d3fd8d9682ddb6c1421828637a04282d"} Nov 02 15:27:50.243744 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.243730 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" Nov 02 15:27:50.250449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.249527 5476 generic.go:356] "Generic (PLEG): container finished" podID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerID="8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4" exitCode=0 Nov 02 15:27:50.250449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.249593 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" event={"ID":"0f36f5a9-b45a-4c86-986e-3a494915914b","Type":"ContainerDied","Data":"8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4"} Nov 02 15:27:50.250449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.249640 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" event={"ID":"0f36f5a9-b45a-4c86-986e-3a494915914b","Type":"ContainerDied","Data":"0ba38aa0a56460db235623d8d4a0719c86f2b11f0f45c602f4cf0638d3897eed"} Nov 02 15:27:50.250449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.249710 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp" Nov 02 15:27:50.281966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.281932 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.284061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.284038 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx4lp\" (UniqueName: \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-kube-api-access-vx4lp\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.284261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.284227 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.285409 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.285382 5476 scope.go:122] "RemoveContainer" containerID="95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e" Nov 02 15:27:50.285530 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.284390 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.285641 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.285625 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.285741 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.285725 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.285835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.285822 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.287622 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.285913 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.287622 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.285931 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slpsl\" (UniqueName: \"kubernetes.io/projected/0f36f5a9-b45a-4c86-986e-3a494915914b-kube-api-access-slpsl\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.287622 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.285952 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcde6ebe-5ae4-413f-8cfc-d09877e3e297-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.287622 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.285967 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f36f5a9-b45a-4c86-986e-3a494915914b-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.287622 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:27:50.285961 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e\": container with ID starting with 95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e not found: ID does not exist" containerID="95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e" Nov 02 15:27:50.287622 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.286050 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e"} err="failed to get container status \"95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e\": rpc error: code = NotFound desc = could not find container \"95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e\": container with ID starting with 95dc751668408e18d33cc625d5b76a12f6d2dcdfd7b5e1a32f027fd5b189aa1e not found: ID does not exist" Nov 02 15:27:50.287622 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.286074 5476 scope.go:122] "RemoveContainer" containerID="3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc" Nov 02 15:27:50.300202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.300174 5476 scope.go:122] "RemoveContainer" containerID="3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc" Nov 02 15:27:50.300595 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:27:50.300567 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc\": container with ID starting with 3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc not found: ID does not exist" containerID="3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc" Nov 02 15:27:50.300697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.300608 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc"} err="failed to get container status \"3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc\": rpc error: code = NotFound desc = could not find container \"3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc\": container with ID starting with 3d3198c12626aea132a9e91a7fee17b375ebe2e34377361c68cc7ffd4ee4f6bc not found: ID does not exist" Nov 02 15:27:50.300697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.300635 5476 scope.go:122] "RemoveContainer" containerID="df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9" Nov 02 15:27:50.312025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.311974 5476 scope.go:122] "RemoveContainer" containerID="df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9" Nov 02 15:27:50.312460 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:27:50.312433 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9\": container with ID starting with df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9 not found: ID does not exist" containerID="df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9" Nov 02 15:27:50.312570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.312472 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9"} err="failed to get container status \"df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9\": rpc error: code = NotFound desc = could not find container \"df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9\": container with ID starting with df05d02cc292a4332591f4e3d4c973e6c77dc4e6bf5269d976d742149c99eff9 not found: ID does not exist" Nov 02 15:27:50.312570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.312497 5476 scope.go:122] "RemoveContainer" containerID="8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4" Nov 02 15:27:50.323299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.323278 5476 scope.go:122] "RemoveContainer" containerID="8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4" Nov 02 15:27:50.333774 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:27:50.333705 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4\": container with ID starting with 8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4 not found: ID does not exist" containerID="8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4" Nov 02 15:27:50.334159 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.334108 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4"} err="failed to get container status \"8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4\": rpc error: code = NotFound desc = could not find container \"8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4\": container with ID starting with 8c2691754c8e2403ef20f4c10836b109288ddd078bfbc513ec74cb7713c663f4 not found: ID does not exist" Nov 02 15:27:50.587541 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.587507 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp"] Nov 02 15:27:50.603363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.602691 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-pgxxp"] Nov 02 15:27:50.693415 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.693381 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f"] Nov 02 15:27:50.712669 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.712628 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-td55f"] Nov 02 15:27:50.844127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.843989 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx"] Nov 02 15:27:50.847983 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.847950 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-wwhtx"] Nov 02 15:27:50.954437 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.954406 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw"] Nov 02 15:27:50.963663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.963634 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw"] Nov 02 15:27:50.991051 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.990970 5476 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:27:50.991221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:50.991080 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-jkbgw" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:27:51.924590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:51.924554 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" path="/var/lib/kubelet/pods/0f36f5a9-b45a-4c86-986e-3a494915914b/volumes" Nov 02 15:27:51.925136 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:51.925121 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" path="/var/lib/kubelet/pods/60c43ec4-5cda-4a28-93a2-b8c12d45c0e9/volumes" Nov 02 15:27:51.925576 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:51.925562 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" path="/var/lib/kubelet/pods/8458c4f5-ccc7-4444-b053-61dd0820c344/volumes" Nov 02 15:27:51.926324 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:27:51.926311 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" path="/var/lib/kubelet/pods/bcde6ebe-5ae4-413f-8cfc-d09877e3e297/volumes" Nov 02 15:28:04.604486 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:04.599090 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 15:28:04.604486 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:04.604076 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:28:04.680018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:04.676213 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:28:04.737230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:04.737186 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": context deadline exceeded" Nov 02 15:28:04.738860 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:04.738823 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Nov 02 15:28:04.741009 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:04.740958 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:28:04.857077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:04.850203 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:28:04.962020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:04.961414 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:28:05.485142 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:05.485089 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Nov 02 15:28:05.613885 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:05.613814 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:28:05.615432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:05.615407 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:28:05.672022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:05.671090 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" Nov 02 15:28:05.707957 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:05.703102 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:28:06.731366 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:06.729245 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f"] Nov 02 15:28:06.766025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:06.765957 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9d0b05fd-s2p8f"] Nov 02 15:28:07.934681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:07.934646 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd46e27-c109-486b-a58d-eb9e058f2a59" path="/var/lib/kubelet/pods/8bd46e27-c109-486b-a58d-eb9e058f2a59/volumes" Nov 02 15:28:14.795696 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.795611 5476 generic.go:356] "Generic (PLEG): container finished" podID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerID="a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462" exitCode=0 Nov 02 15:28:14.796167 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.795686 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" event={"ID":"be7d2cc8-ea14-45c9-b096-578619c36bf0","Type":"ContainerDied","Data":"a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462"} Nov 02 15:28:14.796235 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.796185 5476 scope.go:122] "RemoveContainer" containerID="a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462" Nov 02 15:28:14.797464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.797445 5476 generic.go:356] "Generic (PLEG): container finished" podID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerID="c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd" exitCode=0 Nov 02 15:28:14.797561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.797527 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" event={"ID":"c3e6b2ff-a84f-428d-9df4-cb020df76d2f","Type":"ContainerDied","Data":"c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd"} Nov 02 15:28:14.797832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.797804 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:14.797915 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.797873 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:14.797976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.797915 5476 scope.go:122] "RemoveContainer" containerID="c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd" Nov 02 15:28:14.798053 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.797917 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:14.799403 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.799366 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:14.799482 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.799433 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:14.799482 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:14.799474 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:28:15.833608 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:15.833573 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" event={"ID":"c3e6b2ff-a84f-428d-9df4-cb020df76d2f","Type":"ContainerStarted","Data":"052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3"} Nov 02 15:28:15.836116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:15.836086 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" event={"ID":"be7d2cc8-ea14-45c9-b096-578619c36bf0","Type":"ContainerStarted","Data":"11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42"} Nov 02 15:28:15.921021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:15.920435 5476 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:28:15.921021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:15.920512 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:28:15.930009 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:15.929925 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:28:16.294019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:16.293205 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:28:16.294019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:16.293694 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 15:28:16.294019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:16.293767 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 15:28:16.918021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:16.910401 5476 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:28:16.918021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:16.910464 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:28:17.317859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:17.317670 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 15:28:17.318325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:17.318099 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 15:28:17.952059 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:17.948604 5476 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:28:17.952059 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:17.948664 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:28:18.345079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.335796 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 15:28:18.345079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.335856 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 15:28:18.431178 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.431137 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:28:18.437044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.437016 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:28:18.663538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.663497 5476 scope.go:122] "RemoveContainer" containerID="e90ca589223ce5c7036b7d9039ada9746e8d18bd3fd6a5bd80219501682268fb" Nov 02 15:28:18.697813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.697788 5476 scope.go:122] "RemoveContainer" containerID="c0ec3e97d8f92bda094a9db04edafd9d9502896456ae51811c029a2af9c64572" Nov 02 15:28:18.740771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.740124 5476 scope.go:122] "RemoveContainer" containerID="6c944e37da924a1d7e5e18f8b15ff4afe401e322b42e9518ac786b9f59c1d1f5" Nov 02 15:28:18.754118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.754091 5476 scope.go:122] "RemoveContainer" containerID="2b76b5913fb5a2a6b540394dd6504637c15a7c9fae5d9e5edec5323db034892d" Nov 02 15:28:18.763889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.763856 5476 scope.go:122] "RemoveContainer" containerID="08afe708cf0a6b6dd2782bccb2666822a23eb0aec567960bfeb623b907916616" Nov 02 15:28:18.828766 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.828744 5476 scope.go:122] "RemoveContainer" containerID="b4aa070bf6e2411cb2fc31deafa2850d01fe54577977658f5cebb9549663b7d4" Nov 02 15:28:18.839965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.839944 5476 scope.go:122] "RemoveContainer" containerID="27278495d214903f8e313956bd32482b9deb4802ca44b3a65bd734a2f598e50b" Nov 02 15:28:18.921056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.905097 5476 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:28:18.921056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:18.905216 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:28:19.294069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:19.293420 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 15:28:19.294069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:19.293502 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 15:28:19.903853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:19.903385 5476 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:28:19.903853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:19.903461 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:28:20.147547 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:28:20.147509 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:28:20.316074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:20.315115 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 15:28:20.316074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:20.315188 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 15:28:20.903476 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:20.903433 5476 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 15:28:20.903788 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:20.903755 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 15:28:21.295080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:21.292365 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 02 15:28:21.295080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:21.292427 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 02 15:28:21.595588 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:21.595498 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:28:21.906374 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:21.906346 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:28:22.176563 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:22.176488 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:28:22.298076 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:22.298047 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:28:22.298394 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:22.298373 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:28:22.299471 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:22.299445 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:28:32.851176 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851098 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv"] Nov 02 15:28:32.851756 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851739 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ccea56fd-e01e-4948-b2a5-1bc7d0c263bd" containerName="revoke-keys" Nov 02 15:28:32.851801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851757 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccea56fd-e01e-4948-b2a5-1bc7d0c263bd" containerName="revoke-keys" Nov 02 15:28:32.851801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851772 5476 cpu_manager.go:414] "containerMap: removing container" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" Nov 02 15:28:32.851801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851778 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" Nov 02 15:28:32.851801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851795 5476 cpu_manager.go:414] "containerMap: removing container" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" Nov 02 15:28:32.851801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851800 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" Nov 02 15:28:32.851952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851818 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" Nov 02 15:28:32.851952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851822 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" Nov 02 15:28:32.851952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851841 5476 cpu_manager.go:414] "containerMap: removing container" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" Nov 02 15:28:32.851952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851845 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" Nov 02 15:28:32.851952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851851 5476 cpu_manager.go:414] "containerMap: removing container" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" Nov 02 15:28:32.851952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851856 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" Nov 02 15:28:32.851952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851940 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="8458c4f5-ccc7-4444-b053-61dd0820c344" containerName="istio-proxy" Nov 02 15:28:32.851952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851949 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcde6ebe-5ae4-413f-8cfc-d09877e3e297" containerName="istio-proxy" Nov 02 15:28:32.852229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851957 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccea56fd-e01e-4948-b2a5-1bc7d0c263bd" containerName="revoke-keys" Nov 02 15:28:32.852229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851964 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c43ec4-5cda-4a28-93a2-b8c12d45c0e9" containerName="istio-proxy" Nov 02 15:28:32.852229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851971 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f36f5a9-b45a-4c86-986e-3a494915914b" containerName="istio-proxy" Nov 02 15:28:32.852229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.851980 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c27ff3-82e0-4411-ab70-c2189b9cf795" containerName="istio-proxy" Nov 02 15:28:32.857699 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.857678 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:32.905538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.905505 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:32.905665 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.905646 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwtqr\" (UniqueName: \"kubernetes.io/projected/c64d4b49-85ca-4a22-8711-c335b8046988-kube-api-access-wwtqr\") pod \"maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:32.905731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.905716 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:32.944008 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.943966 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:28:32.952557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.952525 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:28:32.956330 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:32.956304 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv"] Nov 02 15:28:33.006720 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:33.006691 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwtqr\" (UniqueName: \"kubernetes.io/projected/c64d4b49-85ca-4a22-8711-c335b8046988-kube-api-access-wwtqr\") pod \"maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:33.006871 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:33.006758 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:33.006937 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:33.006925 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:33.007538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:33.007513 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:33.009256 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:33.009235 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:33.069907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:33.069883 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwtqr\" (UniqueName: \"kubernetes.io/projected/c64d4b49-85ca-4a22-8711-c335b8046988-kube-api-access-wwtqr\") pod \"maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:33.171688 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:33.171660 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:33.552268 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:33.552193 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv"] Nov 02 15:28:33.556271 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:28:33.556243 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64d4b49_85ca_4a22_8711_c335b8046988.slice/crio-f4461b7d9aead2df2ffecf019ea64ae5fea66a62d5b5d4a71521844666463527 WatchSource:0}: Error finding container f4461b7d9aead2df2ffecf019ea64ae5fea66a62d5b5d4a71521844666463527: Status 404 returned error can't find the container with id f4461b7d9aead2df2ffecf019ea64ae5fea66a62d5b5d4a71521844666463527 Nov 02 15:28:34.266784 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:34.266690 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" event={"ID":"c64d4b49-85ca-4a22-8711-c335b8046988","Type":"ContainerStarted","Data":"faf33abf1997ef00bfc6fc7824f6a8ad6e397dfb2f5590e1eb766ec8b27973db"} Nov 02 15:28:34.266784 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:34.266755 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" event={"ID":"c64d4b49-85ca-4a22-8711-c335b8046988","Type":"ContainerStarted","Data":"f4461b7d9aead2df2ffecf019ea64ae5fea66a62d5b5d4a71521844666463527"} Nov 02 15:28:34.387111 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:34.387005 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" podStartSLOduration=2.38697876 podStartE2EDuration="2.38697876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:34.383821505 +0000 UTC m=+1816.586699578" watchObservedRunningTime="2026-11-02 15:28:34.38697876 +0000 UTC m=+1816.589856858" Nov 02 15:28:35.275687 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:35.275644 5476 generic.go:356] "Generic (PLEG): container finished" podID="c64d4b49-85ca-4a22-8711-c335b8046988" containerID="faf33abf1997ef00bfc6fc7824f6a8ad6e397dfb2f5590e1eb766ec8b27973db" exitCode=0 Nov 02 15:28:35.276204 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:35.275705 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" event={"ID":"c64d4b49-85ca-4a22-8711-c335b8046988","Type":"ContainerDied","Data":"faf33abf1997ef00bfc6fc7824f6a8ad6e397dfb2f5590e1eb766ec8b27973db"} Nov 02 15:28:36.454329 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:36.454296 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:36.560611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:36.560516 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-openshift-ca\") pod \"c64d4b49-85ca-4a22-8711-c335b8046988\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " Nov 02 15:28:36.561111 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:36.561080 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64d4b49-85ca-4a22-8711-c335b8046988-kube-api-access-wwtqr\" (UniqueName: \"kubernetes.io/projected/c64d4b49-85ca-4a22-8711-c335b8046988-kube-api-access-wwtqr\") pod \"c64d4b49-85ca-4a22-8711-c335b8046988\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " Nov 02 15:28:36.562178 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:36.562124 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-xks-ca\") pod \"c64d4b49-85ca-4a22-8711-c335b8046988\" (UID: \"c64d4b49-85ca-4a22-8711-c335b8046988\") " Nov 02 15:28:36.563331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:36.563283 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-openshift-ca" pod "c64d4b49-85ca-4a22-8711-c335b8046988" (UID: "c64d4b49-85ca-4a22-8711-c335b8046988"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:28:36.564528 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:36.564508 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:36.597849 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:36.596445 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64d4b49-85ca-4a22-8711-c335b8046988-kube-api-access-wwtqr" pod "c64d4b49-85ca-4a22-8711-c335b8046988" (UID: "c64d4b49-85ca-4a22-8711-c335b8046988"). InnerVolumeSpecName "kube-api-access-wwtqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:28:36.610704 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:36.610638 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-xks-ca" pod "c64d4b49-85ca-4a22-8711-c335b8046988" (UID: "c64d4b49-85ca-4a22-8711-c335b8046988"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:28:36.689990 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:36.689927 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64d4b49-85ca-4a22-8711-c335b8046988-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:36.689990 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:36.689964 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwtqr\" (UniqueName: \"kubernetes.io/projected/c64d4b49-85ca-4a22-8711-c335b8046988-kube-api-access-wwtqr\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:36.930196 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:36.930164 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:28:37.291441 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:37.291355 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" Nov 02 15:28:37.291441 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:37.291392 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv" event={"ID":"c64d4b49-85ca-4a22-8711-c335b8046988","Type":"ContainerDied","Data":"f4461b7d9aead2df2ffecf019ea64ae5fea66a62d5b5d4a71521844666463527"} Nov 02 15:28:37.291625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:28:37.291450 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4461b7d9aead2df2ffecf019ea64ae5fea66a62d5b5d4a71521844666463527" Nov 02 15:28:50.147420 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:28:50.147373 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:29:15.616625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:15.616586 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv"] Nov 02 15:29:15.660215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:15.660181 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5b52693-ztgmv"] Nov 02 15:29:15.924531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:15.924454 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64d4b49-85ca-4a22-8711-c335b8046988" path="/var/lib/kubelet/pods/c64d4b49-85ca-4a22-8711-c335b8046988/volumes" Nov 02 15:29:20.147441 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:29:20.147406 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:29:29.625453 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:29.625416 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96"] Nov 02 15:29:29.626597 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:29.625627 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerName="istio-proxy" containerID="cri-o://3bb713b61af2b2d6c3803c3ae456ac6d5bdae6eb75d4b5c4ca56aa49d8bc4b74" gracePeriod=30 Nov 02 15:29:29.673181 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:29.673145 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9"] Nov 02 15:29:29.673412 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:29.673386 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" podUID="a1549618-289f-426e-997d-277d2705d0db" containerName="istio-proxy" containerID="cri-o://20f794160c7907d364c14cff8912f5f965d816f8fab7ca05823af7a57a32ad64" gracePeriod=30 Nov 02 15:29:29.678127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:29.678095 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l"] Nov 02 15:29:29.678351 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:29.678319 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" containerID="cri-o://0c68d041a88c13b28c52071cc4ef668f1fef398b2caa90bab20652bf7332bdb4" gracePeriod=30 Nov 02 15:29:29.716471 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:29.716431 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8"] Nov 02 15:29:29.716654 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:29.716629 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" containerID="cri-o://8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb" gracePeriod=30 Nov 02 15:29:33.571019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:33.570441 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:29:33.571019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:33.570513 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:29:34.449636 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:34.449594 5476 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:29:34.449814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:34.449657 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:29:34.829906 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:34.829870 5476 generic.go:356] "Generic (PLEG): container finished" podID="496f7b65-1081-4742-a595-414339df2dee" containerID="0c68d041a88c13b28c52071cc4ef668f1fef398b2caa90bab20652bf7332bdb4" exitCode=0 Nov 02 15:29:34.830417 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:34.830038 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" event={"ID":"496f7b65-1081-4742-a595-414339df2dee","Type":"ContainerDied","Data":"0c68d041a88c13b28c52071cc4ef668f1fef398b2caa90bab20652bf7332bdb4"} Nov 02 15:29:34.833005 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:34.832966 5476 generic.go:356] "Generic (PLEG): container finished" podID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerID="3bb713b61af2b2d6c3803c3ae456ac6d5bdae6eb75d4b5c4ca56aa49d8bc4b74" exitCode=0 Nov 02 15:29:34.833151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:34.833131 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" event={"ID":"80fe2e6c-e885-4e1d-b974-54a98225dad1","Type":"ContainerDied","Data":"3bb713b61af2b2d6c3803c3ae456ac6d5bdae6eb75d4b5c4ca56aa49d8bc4b74"} Nov 02 15:29:34.837120 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:34.836136 5476 generic.go:356] "Generic (PLEG): container finished" podID="a1549618-289f-426e-997d-277d2705d0db" containerID="20f794160c7907d364c14cff8912f5f965d816f8fab7ca05823af7a57a32ad64" exitCode=0 Nov 02 15:29:34.837120 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:34.836174 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" event={"ID":"a1549618-289f-426e-997d-277d2705d0db","Type":"ContainerDied","Data":"20f794160c7907d364c14cff8912f5f965d816f8fab7ca05823af7a57a32ad64"} Nov 02 15:29:35.091657 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.090884 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:29:35.103142 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.103114 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:29:35.112916 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.112894 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:29:35.116627 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.116606 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:29:35.123082 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123061 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-podinfo\") pod \"6c9cb32e-03ea-4d07-8763-0b299a061334\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " Nov 02 15:29:35.123167 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123102 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-socket\") pod \"6c9cb32e-03ea-4d07-8763-0b299a061334\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " Nov 02 15:29:35.123167 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123122 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-envoy\") pod \"80fe2e6c-e885-4e1d-b974-54a98225dad1\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " Nov 02 15:29:35.123167 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123140 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-token\" (UniqueName: \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-token\") pod \"6c9cb32e-03ea-4d07-8763-0b299a061334\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " Nov 02 15:29:35.123313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123275 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-envoy\") pod \"496f7b65-1081-4742-a595-414339df2dee\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " Nov 02 15:29:35.123372 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123328 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-data\") pod \"496f7b65-1081-4742-a595-414339df2dee\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " Nov 02 15:29:35.123430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123361 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-kube-api-access-nc89v\" (UniqueName: \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-kube-api-access-nc89v\") pod \"6c9cb32e-03ea-4d07-8763-0b299a061334\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " Nov 02 15:29:35.123430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123364 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-socket" pod "6c9cb32e-03ea-4d07-8763-0b299a061334" (UID: "6c9cb32e-03ea-4d07-8763-0b299a061334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.123430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123399 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-data\") pod \"a1549618-289f-426e-997d-277d2705d0db\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " Nov 02 15:29:35.123590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123467 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-data\") pod \"80fe2e6c-e885-4e1d-b974-54a98225dad1\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " Nov 02 15:29:35.123590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123503 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-socket\") pod \"a1549618-289f-426e-997d-277d2705d0db\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " Nov 02 15:29:35.123590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123568 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1549618-289f-426e-997d-277d2705d0db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1549618-289f-426e-997d-277d2705d0db-istio-podinfo\") pod \"a1549618-289f-426e-997d-277d2705d0db\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " Nov 02 15:29:35.123741 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123581 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-data" pod "496f7b65-1081-4742-a595-414339df2dee" (UID: "496f7b65-1081-4742-a595-414339df2dee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.123741 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123600 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-istio-token\" (UniqueName: \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-istio-token\") pod \"a1549618-289f-426e-997d-277d2705d0db\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " Nov 02 15:29:35.123741 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123629 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-envoy\") pod \"a1549618-289f-426e-997d-277d2705d0db\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " Nov 02 15:29:35.123741 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123659 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1549618-289f-426e-997d-277d2705d0db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1549618-289f-426e-997d-277d2705d0db-istiod-ca-cert\") pod \"a1549618-289f-426e-997d-277d2705d0db\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " Nov 02 15:29:35.123741 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123692 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-podinfo\") pod \"80fe2e6c-e885-4e1d-b974-54a98225dad1\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " Nov 02 15:29:35.123741 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123727 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-data" pod "a1549618-289f-426e-997d-277d2705d0db" (UID: "a1549618-289f-426e-997d-277d2705d0db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.124154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123729 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80fe2e6c-e885-4e1d-b974-54a98225dad1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fe2e6c-e885-4e1d-b974-54a98225dad1-istiod-ca-cert\") pod \"80fe2e6c-e885-4e1d-b974-54a98225dad1\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " Nov 02 15:29:35.124154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123774 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-socket\") pod \"80fe2e6c-e885-4e1d-b974-54a98225dad1\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " Nov 02 15:29:35.124154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123919 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-socket\") pod \"496f7b65-1081-4742-a595-414339df2dee\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " Nov 02 15:29:35.124154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123946 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-certs\") pod \"496f7b65-1081-4742-a595-414339df2dee\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " Nov 02 15:29:35.124154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123973 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-certs\") pod \"6c9cb32e-03ea-4d07-8763-0b299a061334\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " Nov 02 15:29:35.124154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124034 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-kube-api-access-nsnlx\" (UniqueName: \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-kube-api-access-nsnlx\") pod \"496f7b65-1081-4742-a595-414339df2dee\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " Nov 02 15:29:35.124154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124071 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/496f7b65-1081-4742-a595-414339df2dee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496f7b65-1081-4742-a595-414339df2dee-istio-podinfo\") pod \"496f7b65-1081-4742-a595-414339df2dee\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " Nov 02 15:29:35.124154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124099 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-token\" (UniqueName: \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-token\") pod \"80fe2e6c-e885-4e1d-b974-54a98225dad1\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " Nov 02 15:29:35.124575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124172 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-certs\") pod \"80fe2e6c-e885-4e1d-b974-54a98225dad1\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " Nov 02 15:29:35.124575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124204 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-kube-api-access-7l55h\" (UniqueName: \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-kube-api-access-7l55h\") pod \"a1549618-289f-426e-997d-277d2705d0db\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " Nov 02 15:29:35.124575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124226 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-credential-socket\") pod \"6c9cb32e-03ea-4d07-8763-0b299a061334\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " Nov 02 15:29:35.124575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124247 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-certs\") pod \"a1549618-289f-426e-997d-277d2705d0db\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " Nov 02 15:29:35.124575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124277 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9cb32e-03ea-4d07-8763-0b299a061334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9cb32e-03ea-4d07-8763-0b299a061334-istiod-ca-cert\") pod \"6c9cb32e-03ea-4d07-8763-0b299a061334\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " Nov 02 15:29:35.124575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124306 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-istio-token\" (UniqueName: \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-istio-token\") pod \"496f7b65-1081-4742-a595-414339df2dee\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " Nov 02 15:29:35.124575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124342 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-kube-api-access-mnhfn\" (UniqueName: \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-kube-api-access-mnhfn\") pod \"80fe2e6c-e885-4e1d-b974-54a98225dad1\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " Nov 02 15:29:35.124575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124398 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-data\") pod \"6c9cb32e-03ea-4d07-8763-0b299a061334\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " Nov 02 15:29:35.124575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124453 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-credential-socket\") pod \"a1549618-289f-426e-997d-277d2705d0db\" (UID: \"a1549618-289f-426e-997d-277d2705d0db\") " Nov 02 15:29:35.124575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124501 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-credential-socket\") pod \"496f7b65-1081-4742-a595-414339df2dee\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " Nov 02 15:29:35.124575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124553 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-credential-socket\") pod \"80fe2e6c-e885-4e1d-b974-54a98225dad1\" (UID: \"80fe2e6c-e885-4e1d-b974-54a98225dad1\") " Nov 02 15:29:35.125203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124589 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496f7b65-1081-4742-a595-414339df2dee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496f7b65-1081-4742-a595-414339df2dee-istiod-ca-cert\") pod \"496f7b65-1081-4742-a595-414339df2dee\" (UID: \"496f7b65-1081-4742-a595-414339df2dee\") " Nov 02 15:29:35.125203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124634 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-envoy\") pod \"6c9cb32e-03ea-4d07-8763-0b299a061334\" (UID: \"6c9cb32e-03ea-4d07-8763-0b299a061334\") " Nov 02 15:29:35.126228 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.125894 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.126228 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.125920 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.126228 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.125937 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.126492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.123840 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-socket" pod "a1549618-289f-426e-997d-277d2705d0db" (UID: "a1549618-289f-426e-997d-277d2705d0db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.126492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124046 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-data" pod "80fe2e6c-e885-4e1d-b974-54a98225dad1" (UID: "80fe2e6c-e885-4e1d-b974-54a98225dad1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.126492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124194 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fe2e6c-e885-4e1d-b974-54a98225dad1-istiod-ca-cert" pod "80fe2e6c-e885-4e1d-b974-54a98225dad1" (UID: "80fe2e6c-e885-4e1d-b974-54a98225dad1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:35.126492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.124538 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1549618-289f-426e-997d-277d2705d0db-istiod-ca-cert" pod "a1549618-289f-426e-997d-277d2705d0db" (UID: "a1549618-289f-426e-997d-277d2705d0db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:35.126492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.125869 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-podinfo" pod "6c9cb32e-03ea-4d07-8763-0b299a061334" (UID: "6c9cb32e-03ea-4d07-8763-0b299a061334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:35.126492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.125884 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-certs" pod "a1549618-289f-426e-997d-277d2705d0db" (UID: "a1549618-289f-426e-997d-277d2705d0db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.126492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.126065 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-credential-socket" pod "6c9cb32e-03ea-4d07-8763-0b299a061334" (UID: "6c9cb32e-03ea-4d07-8763-0b299a061334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.126492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.126108 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-certs" pod "6c9cb32e-03ea-4d07-8763-0b299a061334" (UID: "6c9cb32e-03ea-4d07-8763-0b299a061334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.126492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.126297 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-socket" pod "80fe2e6c-e885-4e1d-b974-54a98225dad1" (UID: "80fe2e6c-e885-4e1d-b974-54a98225dad1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.126492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.126342 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-credential-socket" pod "496f7b65-1081-4742-a595-414339df2dee" (UID: "496f7b65-1081-4742-a595-414339df2dee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.126492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.126391 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-data" pod "6c9cb32e-03ea-4d07-8763-0b299a061334" (UID: "6c9cb32e-03ea-4d07-8763-0b299a061334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.127182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.126553 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-socket" pod "496f7b65-1081-4742-a595-414339df2dee" (UID: "496f7b65-1081-4742-a595-414339df2dee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.127182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.126575 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-token" pod "6c9cb32e-03ea-4d07-8763-0b299a061334" (UID: "6c9cb32e-03ea-4d07-8763-0b299a061334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:35.127182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.126644 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-envoy" pod "80fe2e6c-e885-4e1d-b974-54a98225dad1" (UID: "80fe2e6c-e885-4e1d-b974-54a98225dad1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.127182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.126743 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-certs" pod "496f7b65-1081-4742-a595-414339df2dee" (UID: "496f7b65-1081-4742-a595-414339df2dee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.127182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.126747 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-kube-api-access-nc89v" pod "6c9cb32e-03ea-4d07-8763-0b299a061334" (UID: "6c9cb32e-03ea-4d07-8763-0b299a061334"). InnerVolumeSpecName "kube-api-access-nc89v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:35.127487 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.127195 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-credential-socket" pod "80fe2e6c-e885-4e1d-b974-54a98225dad1" (UID: "80fe2e6c-e885-4e1d-b974-54a98225dad1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.127487 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.127227 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9cb32e-03ea-4d07-8763-0b299a061334-istiod-ca-cert" pod "6c9cb32e-03ea-4d07-8763-0b299a061334" (UID: "6c9cb32e-03ea-4d07-8763-0b299a061334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:35.127604 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.127530 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496f7b65-1081-4742-a595-414339df2dee-istiod-ca-cert" pod "496f7b65-1081-4742-a595-414339df2dee" (UID: "496f7b65-1081-4742-a595-414339df2dee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:35.127666 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.127612 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-credential-socket" pod "a1549618-289f-426e-997d-277d2705d0db" (UID: "a1549618-289f-426e-997d-277d2705d0db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.127824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.127791 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-certs" pod "80fe2e6c-e885-4e1d-b974-54a98225dad1" (UID: "80fe2e6c-e885-4e1d-b974-54a98225dad1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.127883 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.127845 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-envoy" pod "a1549618-289f-426e-997d-277d2705d0db" (UID: "a1549618-289f-426e-997d-277d2705d0db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.128063 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.128041 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-podinfo" pod "80fe2e6c-e885-4e1d-b974-54a98225dad1" (UID: "80fe2e6c-e885-4e1d-b974-54a98225dad1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:35.128357 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.128329 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-envoy" pod "496f7b65-1081-4742-a595-414339df2dee" (UID: "496f7b65-1081-4742-a595-414339df2dee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.129088 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.128971 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-kube-api-access-7l55h" pod "a1549618-289f-426e-997d-277d2705d0db" (UID: "a1549618-289f-426e-997d-277d2705d0db"). InnerVolumeSpecName "kube-api-access-7l55h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:35.129088 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.129060 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-envoy" pod "6c9cb32e-03ea-4d07-8763-0b299a061334" (UID: "6c9cb32e-03ea-4d07-8763-0b299a061334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:35.129365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.129327 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-istio-token" pod "a1549618-289f-426e-997d-277d2705d0db" (UID: "a1549618-289f-426e-997d-277d2705d0db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:35.129921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.129892 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/496f7b65-1081-4742-a595-414339df2dee-istio-podinfo" pod "496f7b65-1081-4742-a595-414339df2dee" (UID: "496f7b65-1081-4742-a595-414339df2dee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:35.130069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.129936 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-kube-api-access-nsnlx" pod "496f7b65-1081-4742-a595-414339df2dee" (UID: "496f7b65-1081-4742-a595-414339df2dee"). InnerVolumeSpecName "kube-api-access-nsnlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:35.130069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.129966 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-token" pod "80fe2e6c-e885-4e1d-b974-54a98225dad1" (UID: "80fe2e6c-e885-4e1d-b974-54a98225dad1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:35.130552 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.130531 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-istio-token" pod "496f7b65-1081-4742-a595-414339df2dee" (UID: "496f7b65-1081-4742-a595-414339df2dee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:35.130641 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.130551 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1549618-289f-426e-997d-277d2705d0db-istio-podinfo" pod "a1549618-289f-426e-997d-277d2705d0db" (UID: "a1549618-289f-426e-997d-277d2705d0db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:35.130985 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.130960 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-kube-api-access-mnhfn" pod "80fe2e6c-e885-4e1d-b974-54a98225dad1" (UID: "80fe2e6c-e885-4e1d-b974-54a98225dad1"). InnerVolumeSpecName "kube-api-access-mnhfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:35.227312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227225 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227256 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227270 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227286 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496f7b65-1081-4742-a595-414339df2dee-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227296 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227305 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227315 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227323 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227332 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227345 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc89v\" (UniqueName: \"kubernetes.io/projected/6c9cb32e-03ea-4d07-8763-0b299a061334-kube-api-access-nc89v\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227359 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227367 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227375 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1549618-289f-426e-997d-277d2705d0db-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227383 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227391 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227399 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1549618-289f-426e-997d-277d2705d0db-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227407 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227418 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fe2e6c-e885-4e1d-b974-54a98225dad1-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227429 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227439 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227447 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496f7b65-1081-4742-a595-414339df2dee-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227455 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227463 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsnlx\" (UniqueName: \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-kube-api-access-nsnlx\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227472 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496f7b65-1081-4742-a595-414339df2dee-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227480 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227489 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fe2e6c-e885-4e1d-b974-54a98225dad1-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227502 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l55h\" (UniqueName: \"kubernetes.io/projected/a1549618-289f-426e-997d-277d2705d0db-kube-api-access-7l55h\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227514 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227522 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1549618-289f-426e-997d-277d2705d0db-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227530 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9cb32e-03ea-4d07-8763-0b299a061334-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227537 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496f7b65-1081-4742-a595-414339df2dee-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227547 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhfn\" (UniqueName: \"kubernetes.io/projected/80fe2e6c-e885-4e1d-b974-54a98225dad1-kube-api-access-mnhfn\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.227648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.227555 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9cb32e-03ea-4d07-8763-0b299a061334-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:35.846803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.846771 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" Nov 02 15:29:35.847331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.846766 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l" event={"ID":"496f7b65-1081-4742-a595-414339df2dee","Type":"ContainerDied","Data":"cf2a2e459ad8d8e8fe56f03bb4e6cecb6d756d6bbf217b0e4e7a6cf06c1b4f4d"} Nov 02 15:29:35.847331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.846923 5476 scope.go:122] "RemoveContainer" containerID="0c68d041a88c13b28c52071cc4ef668f1fef398b2caa90bab20652bf7332bdb4" Nov 02 15:29:35.848394 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.848369 5476 generic.go:356] "Generic (PLEG): container finished" podID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerID="8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb" exitCode=0 Nov 02 15:29:35.848486 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.848433 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" Nov 02 15:29:35.848538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.848512 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" event={"ID":"6c9cb32e-03ea-4d07-8763-0b299a061334","Type":"ContainerDied","Data":"8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb"} Nov 02 15:29:35.848598 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.848561 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8" event={"ID":"6c9cb32e-03ea-4d07-8763-0b299a061334","Type":"ContainerDied","Data":"8610b865e46796320e7073b297e0ef941ae2c2fb29b5cd8342446e5090893051"} Nov 02 15:29:35.850403 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.850371 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" Nov 02 15:29:35.850540 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.850414 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96" event={"ID":"80fe2e6c-e885-4e1d-b974-54a98225dad1","Type":"ContainerDied","Data":"d838255548c8299f2fcff0de9ddb82884f14903d5826c7dc1ca841378c156cd4"} Nov 02 15:29:35.852029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.852007 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" Nov 02 15:29:35.852029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.852018 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9" event={"ID":"a1549618-289f-426e-997d-277d2705d0db","Type":"ContainerDied","Data":"8e3da0505bf6cba760dc64221a8cbfe96dc892cd13b0a1cd55acde7bb9b161c2"} Nov 02 15:29:35.860745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.860730 5476 scope.go:122] "RemoveContainer" containerID="8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb" Nov 02 15:29:35.871376 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.871357 5476 scope.go:122] "RemoveContainer" containerID="8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb" Nov 02 15:29:35.871646 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:29:35.871630 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb\": container with ID starting with 8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb not found: ID does not exist" containerID="8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb" Nov 02 15:29:35.871696 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.871655 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb"} err="failed to get container status \"8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb\": rpc error: code = NotFound desc = could not find container \"8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb\": container with ID starting with 8f49620d86f52fbe463fc49c437ddfc290277ae83ba97fa5edc31a947ebd4ceb not found: ID does not exist" Nov 02 15:29:35.871696 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.871672 5476 scope.go:122] "RemoveContainer" containerID="3bb713b61af2b2d6c3803c3ae456ac6d5bdae6eb75d4b5c4ca56aa49d8bc4b74" Nov 02 15:29:35.880616 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:35.880598 5476 scope.go:122] "RemoveContainer" containerID="20f794160c7907d364c14cff8912f5f965d816f8fab7ca05823af7a57a32ad64" Nov 02 15:29:36.004333 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:36.004218 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96"] Nov 02 15:29:36.010264 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:36.010237 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-zbl96"] Nov 02 15:29:36.111325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:36.111291 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l"] Nov 02 15:29:36.116564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:36.116537 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-5qc9l"] Nov 02 15:29:36.219220 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:36.219190 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9"] Nov 02 15:29:36.225698 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:36.225670 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-fhzw9"] Nov 02 15:29:36.331066 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:36.330952 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8"] Nov 02 15:29:36.334968 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:36.334938 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vmbb8"] Nov 02 15:29:37.924626 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:37.924592 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496f7b65-1081-4742-a595-414339df2dee" path="/var/lib/kubelet/pods/496f7b65-1081-4742-a595-414339df2dee/volumes" Nov 02 15:29:37.925130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:37.925115 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" path="/var/lib/kubelet/pods/6c9cb32e-03ea-4d07-8763-0b299a061334/volumes" Nov 02 15:29:37.925565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:37.925552 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" path="/var/lib/kubelet/pods/80fe2e6c-e885-4e1d-b974-54a98225dad1/volumes" Nov 02 15:29:37.926292 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:37.926277 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1549618-289f-426e-997d-277d2705d0db" path="/var/lib/kubelet/pods/a1549618-289f-426e-997d-277d2705d0db/volumes" Nov 02 15:29:39.961029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:39.960951 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:29:40.320575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.320373 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr"] Nov 02 15:29:40.321158 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321134 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c64d4b49-85ca-4a22-8711-c335b8046988" containerName="revoke-keys" Nov 02 15:29:40.321158 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321153 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64d4b49-85ca-4a22-8711-c335b8046988" containerName="revoke-keys" Nov 02 15:29:40.321353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321172 5476 cpu_manager.go:414] "containerMap: removing container" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerName="istio-proxy" Nov 02 15:29:40.321353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321177 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerName="istio-proxy" Nov 02 15:29:40.321353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321190 5476 cpu_manager.go:414] "containerMap: removing container" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" Nov 02 15:29:40.321353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321195 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" Nov 02 15:29:40.321353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321211 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" Nov 02 15:29:40.321353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321215 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" Nov 02 15:29:40.321353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321235 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a1549618-289f-426e-997d-277d2705d0db" containerName="istio-proxy" Nov 02 15:29:40.321353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321240 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1549618-289f-426e-997d-277d2705d0db" containerName="istio-proxy" Nov 02 15:29:40.321353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321335 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9cb32e-03ea-4d07-8763-0b299a061334" containerName="istio-proxy" Nov 02 15:29:40.321353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321347 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1549618-289f-426e-997d-277d2705d0db" containerName="istio-proxy" Nov 02 15:29:40.321353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321356 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64d4b49-85ca-4a22-8711-c335b8046988" containerName="revoke-keys" Nov 02 15:29:40.321857 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321363 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="80fe2e6c-e885-4e1d-b974-54a98225dad1" containerName="istio-proxy" Nov 02 15:29:40.321857 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.321370 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="496f7b65-1081-4742-a595-414339df2dee" containerName="istio-proxy" Nov 02 15:29:40.324885 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.324859 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.391021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.386377 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0de358-openshift-default-dockercfg-h66gx\"" Nov 02 15:29:40.416020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.415153 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-credential-socket\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.416020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.415223 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-istio-token\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.416020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.415258 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-socket\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.416020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.415316 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-data\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.416020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.415394 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af3940e-fa13-4569-9951-a338c215ba0d-istiod-ca-cert\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.416020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.415431 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af3940e-fa13-4569-9951-a338c215ba0d-istio-podinfo\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.416020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.415557 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-certs\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.416020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.415614 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkbh6\" (UniqueName: \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-kube-api-access-fkbh6\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.416020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.415674 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-envoy\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.450714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.450660 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr"] Nov 02 15:29:40.534611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.525128 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-credential-socket\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.534611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.525205 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-istio-token\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.534611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.525237 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-socket\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.534611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.525284 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-data\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.534611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.525361 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af3940e-fa13-4569-9951-a338c215ba0d-istiod-ca-cert\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.534611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.525398 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af3940e-fa13-4569-9951-a338c215ba0d-istio-podinfo\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.534611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.525532 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-certs\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.534611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.525600 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkbh6\" (UniqueName: \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-kube-api-access-fkbh6\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.534611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.525664 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-envoy\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.540306 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.539892 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-certs\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.545080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.540511 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-credential-socket\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.549172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.546978 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-socket\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.549172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.549018 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af3940e-fa13-4569-9951-a338c215ba0d-istiod-ca-cert\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.558368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.556310 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-data\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.558368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.557523 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-envoy\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.568120 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.566513 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af3940e-fa13-4569-9951-a338c215ba0d-istio-podinfo\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.599664 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.599594 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-istio-token\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.616579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.616544 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkbh6\" (UniqueName: \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-kube-api-access-fkbh6\") pod \"e2e-api-a-0de358-openshift-default-7466f54655-gnmcr\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:40.663041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:40.662979 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:41.205118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:41.205076 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr"] Nov 02 15:29:41.210516 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:29:41.210470 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af3940e_fa13_4569_9951_a338c215ba0d.slice/crio-7606626f7268271a8a11f53516c4dc0ce7a326fe1038621a78c4e4683b2b5115 WatchSource:0}: Error finding container 7606626f7268271a8a11f53516c4dc0ce7a326fe1038621a78c4e4683b2b5115: Status 404 returned error can't find the container with id 7606626f7268271a8a11f53516c4dc0ce7a326fe1038621a78c4e4683b2b5115 Nov 02 15:29:41.213596 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:41.213555 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:41.213684 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:41.213645 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:41.213724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:41.213686 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:29:41.913793 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:41.913753 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" event={"ID":"9af3940e-fa13-4569-9951-a338c215ba0d","Type":"ContainerStarted","Data":"5d57848a937bff863cfed4a1e34767560641123b211c2c3c612c21211555d498"} Nov 02 15:29:41.913988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:41.913814 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" event={"ID":"9af3940e-fa13-4569-9951-a338c215ba0d","Type":"ContainerStarted","Data":"7606626f7268271a8a11f53516c4dc0ce7a326fe1038621a78c4e4683b2b5115"} Nov 02 15:29:42.119688 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:42.119624 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" podStartSLOduration=2.119610414 podStartE2EDuration="2.119610414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:42.073532616 +0000 UTC m=+1884.276410677" watchObservedRunningTime="2026-11-02 15:29:42.119610414 +0000 UTC m=+1884.322488473" Nov 02 15:29:42.664379 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:42.664345 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:42.670196 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:42.670167 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:42.921740 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:42.921655 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:42.922521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:29:42.922500 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:29:50.148454 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:29:50.148410 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:30:00.277259 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.277211 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7"] Nov 02 15:30:00.282304 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.282277 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs"] Nov 02 15:30:00.282473 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.282391 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" Nov 02 15:30:00.286496 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.286475 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5"] Nov 02 15:30:00.286633 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.286587 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" Nov 02 15:30:00.290513 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.290495 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" Nov 02 15:30:00.360355 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.360318 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn48q\" (UniqueName: \"kubernetes.io/projected/2c481a40-2914-4897-9b11-bb147f3cfadf-kube-api-access-qn48q\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7\" (UID: \"2c481a40-2914-4897-9b11-bb147f3cfadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" Nov 02 15:30:00.360630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.360609 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c481a40-2914-4897-9b11-bb147f3cfadf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7\" (UID: \"2c481a40-2914-4897-9b11-bb147f3cfadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" Nov 02 15:30:00.376435 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.376391 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs"] Nov 02 15:30:00.377738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.377715 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:30:00.385276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.385244 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7"] Nov 02 15:30:00.421698 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.421669 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq"] Nov 02 15:30:00.427335 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.427311 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5"] Nov 02 15:30:00.427549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.427427 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" Nov 02 15:30:00.431629 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.431603 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl"] Nov 02 15:30:00.439234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.436609 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" Nov 02 15:30:00.439234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.436827 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv"] Nov 02 15:30:00.441917 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.441898 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" Nov 02 15:30:00.474061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.464985 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqb4j\" (UniqueName: \"kubernetes.io/projected/3573e422-da50-42c9-98b9-3f544c6414e2-kube-api-access-kqb4j\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5\" (UID: \"3573e422-da50-42c9-98b9-3f544c6414e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" Nov 02 15:30:00.474061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.465064 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3573e422-da50-42c9-98b9-3f544c6414e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5\" (UID: \"3573e422-da50-42c9-98b9-3f544c6414e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" Nov 02 15:30:00.474061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.465319 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c481a40-2914-4897-9b11-bb147f3cfadf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7\" (UID: \"2c481a40-2914-4897-9b11-bb147f3cfadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" Nov 02 15:30:00.474061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.465365 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrqgs\" (UniqueName: \"kubernetes.io/projected/13d5e7e0-4be2-4f59-b268-df73ed648efe-kube-api-access-hrqgs\") pod \"maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs\" (UID: \"13d5e7e0-4be2-4f59-b268-df73ed648efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" Nov 02 15:30:00.474061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.465404 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d5e7e0-4be2-4f59-b268-df73ed648efe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs\" (UID: \"13d5e7e0-4be2-4f59-b268-df73ed648efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" Nov 02 15:30:00.474061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.465452 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn48q\" (UniqueName: \"kubernetes.io/projected/2c481a40-2914-4897-9b11-bb147f3cfadf-kube-api-access-qn48q\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7\" (UID: \"2c481a40-2914-4897-9b11-bb147f3cfadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" Nov 02 15:30:00.474061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.466772 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c481a40-2914-4897-9b11-bb147f3cfadf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7\" (UID: \"2c481a40-2914-4897-9b11-bb147f3cfadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" Nov 02 15:30:00.484936 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.484906 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq"] Nov 02 15:30:00.505795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.505761 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv"] Nov 02 15:30:00.510388 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.510352 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl"] Nov 02 15:30:00.546919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.546838 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn48q\" (UniqueName: \"kubernetes.io/projected/2c481a40-2914-4897-9b11-bb147f3cfadf-kube-api-access-qn48q\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7\" (UID: \"2c481a40-2914-4897-9b11-bb147f3cfadf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" Nov 02 15:30:00.574922 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.574891 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqgs\" (UniqueName: \"kubernetes.io/projected/13d5e7e0-4be2-4f59-b268-df73ed648efe-kube-api-access-hrqgs\") pod \"maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs\" (UID: \"13d5e7e0-4be2-4f59-b268-df73ed648efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" Nov 02 15:30:00.575116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.574932 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d5e7e0-4be2-4f59-b268-df73ed648efe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs\" (UID: \"13d5e7e0-4be2-4f59-b268-df73ed648efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" Nov 02 15:30:00.575176 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.575111 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqb4j\" (UniqueName: \"kubernetes.io/projected/3573e422-da50-42c9-98b9-3f544c6414e2-kube-api-access-kqb4j\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5\" (UID: \"3573e422-da50-42c9-98b9-3f544c6414e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" Nov 02 15:30:00.575176 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.575150 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3573e422-da50-42c9-98b9-3f544c6414e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5\" (UID: \"3573e422-da50-42c9-98b9-3f544c6414e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" Nov 02 15:30:00.575269 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.575204 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf577da-6f95-4a29-b4e3-d373978edcc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv\" (UID: \"bdf577da-6f95-4a29-b4e3-d373978edcc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" Nov 02 15:30:00.575269 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.575255 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8kq4\" (UniqueName: \"kubernetes.io/projected/bdf577da-6f95-4a29-b4e3-d373978edcc9-kube-api-access-j8kq4\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv\" (UID: \"bdf577da-6f95-4a29-b4e3-d373978edcc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" Nov 02 15:30:00.575657 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.575629 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5brd9\" (UniqueName: \"kubernetes.io/projected/ead73082-e749-40ec-b4a5-e9583125a9aa-kube-api-access-5brd9\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq\" (UID: \"ead73082-e749-40ec-b4a5-e9583125a9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" Nov 02 15:30:00.575771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.575754 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead73082-e749-40ec-b4a5-e9583125a9aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq\" (UID: \"ead73082-e749-40ec-b4a5-e9583125a9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" Nov 02 15:30:00.575822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.575797 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d5e7e0-4be2-4f59-b268-df73ed648efe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs\" (UID: \"13d5e7e0-4be2-4f59-b268-df73ed648efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" Nov 02 15:30:00.575857 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.575819 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkjxv\" (UniqueName: \"kubernetes.io/projected/be99deda-eff2-4a81-a338-8a683c54d176-kube-api-access-hkjxv\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl\" (UID: \"be99deda-eff2-4a81-a338-8a683c54d176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" Nov 02 15:30:00.575896 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.575888 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be99deda-eff2-4a81-a338-8a683c54d176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl\" (UID: \"be99deda-eff2-4a81-a338-8a683c54d176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" Nov 02 15:30:00.575931 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.575888 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3573e422-da50-42c9-98b9-3f544c6414e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5\" (UID: \"3573e422-da50-42c9-98b9-3f544c6414e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" Nov 02 15:30:00.599215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.599182 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" Nov 02 15:30:00.643221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.643189 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqgs\" (UniqueName: \"kubernetes.io/projected/13d5e7e0-4be2-4f59-b268-df73ed648efe-kube-api-access-hrqgs\") pod \"maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs\" (UID: \"13d5e7e0-4be2-4f59-b268-df73ed648efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" Nov 02 15:30:00.643221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.643210 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqb4j\" (UniqueName: \"kubernetes.io/projected/3573e422-da50-42c9-98b9-3f544c6414e2-kube-api-access-kqb4j\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5\" (UID: \"3573e422-da50-42c9-98b9-3f544c6414e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" Nov 02 15:30:00.688842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.686188 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead73082-e749-40ec-b4a5-e9583125a9aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq\" (UID: \"ead73082-e749-40ec-b4a5-e9583125a9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" Nov 02 15:30:00.688842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.686257 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkjxv\" (UniqueName: \"kubernetes.io/projected/be99deda-eff2-4a81-a338-8a683c54d176-kube-api-access-hkjxv\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl\" (UID: \"be99deda-eff2-4a81-a338-8a683c54d176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" Nov 02 15:30:00.688842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.686365 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be99deda-eff2-4a81-a338-8a683c54d176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl\" (UID: \"be99deda-eff2-4a81-a338-8a683c54d176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" Nov 02 15:30:00.688842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.686550 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf577da-6f95-4a29-b4e3-d373978edcc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv\" (UID: \"bdf577da-6f95-4a29-b4e3-d373978edcc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" Nov 02 15:30:00.688842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.686604 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8kq4\" (UniqueName: \"kubernetes.io/projected/bdf577da-6f95-4a29-b4e3-d373978edcc9-kube-api-access-j8kq4\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv\" (UID: \"bdf577da-6f95-4a29-b4e3-d373978edcc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" Nov 02 15:30:00.688842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.686658 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5brd9\" (UniqueName: \"kubernetes.io/projected/ead73082-e749-40ec-b4a5-e9583125a9aa-kube-api-access-5brd9\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq\" (UID: \"ead73082-e749-40ec-b4a5-e9583125a9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" Nov 02 15:30:00.688842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.687054 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead73082-e749-40ec-b4a5-e9583125a9aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq\" (UID: \"ead73082-e749-40ec-b4a5-e9583125a9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" Nov 02 15:30:00.688842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.687718 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf577da-6f95-4a29-b4e3-d373978edcc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv\" (UID: \"bdf577da-6f95-4a29-b4e3-d373978edcc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" Nov 02 15:30:00.688842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.688065 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be99deda-eff2-4a81-a338-8a683c54d176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl\" (UID: \"be99deda-eff2-4a81-a338-8a683c54d176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" Nov 02 15:30:00.744566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.744529 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5brd9\" (UniqueName: \"kubernetes.io/projected/ead73082-e749-40ec-b4a5-e9583125a9aa-kube-api-access-5brd9\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq\" (UID: \"ead73082-e749-40ec-b4a5-e9583125a9aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" Nov 02 15:30:00.749261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.749232 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkjxv\" (UniqueName: \"kubernetes.io/projected/be99deda-eff2-4a81-a338-8a683c54d176-kube-api-access-hkjxv\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl\" (UID: \"be99deda-eff2-4a81-a338-8a683c54d176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" Nov 02 15:30:00.752792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.752768 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8kq4\" (UniqueName: \"kubernetes.io/projected/bdf577da-6f95-4a29-b4e3-d373978edcc9-kube-api-access-j8kq4\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv\" (UID: \"bdf577da-6f95-4a29-b4e3-d373978edcc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" Nov 02 15:30:00.776509 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.776478 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" Nov 02 15:30:00.791527 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.791503 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" Nov 02 15:30:00.797684 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.797631 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" Nov 02 15:30:00.908554 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.907336 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" Nov 02 15:30:00.913072 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:00.912848 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" Nov 02 15:30:01.044468 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:30:01.043825 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c481a40_2914_4897_9b11_bb147f3cfadf.slice/crio-1e29bfefa9acd43566ca25f9b16b8fee1ad97d6da12580e18da30d59cd401afd WatchSource:0}: Error finding container 1e29bfefa9acd43566ca25f9b16b8fee1ad97d6da12580e18da30d59cd401afd: Status 404 returned error can't find the container with id 1e29bfefa9acd43566ca25f9b16b8fee1ad97d6da12580e18da30d59cd401afd Nov 02 15:30:01.044984 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.044953 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7"] Nov 02 15:30:01.075327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.075282 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4"] Nov 02 15:30:01.081596 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.081568 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" event={"ID":"2c481a40-2914-4897-9b11-bb147f3cfadf","Type":"ContainerStarted","Data":"1e29bfefa9acd43566ca25f9b16b8fee1ad97d6da12580e18da30d59cd401afd"} Nov 02 15:30:01.081706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.081672 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.093825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.093362 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.093825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.093458 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m98z4\" (UniqueName: \"kubernetes.io/projected/ff40bd90-4116-4569-ac57-780ffeee857a-kube-api-access-m98z4\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.093825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.093606 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff40bd90-4116-4569-ac57-780ffeee857a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.093825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.093700 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.143494 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.143465 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ff6e1f-serving-cert\"" Nov 02 15:30:01.166692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.166652 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4"] Nov 02 15:30:01.194713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.194669 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.194909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.194731 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.194909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.194788 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m98z4\" (UniqueName: \"kubernetes.io/projected/ff40bd90-4116-4569-ac57-780ffeee857a-kube-api-access-m98z4\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.194909 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:01.194853 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-serving-cert: secret "maas-api-e2e-shared-b-w2-ff6e1f-serving-cert" not found Nov 02 15:30:01.195127 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:01.194932 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-maas-api-tls podName:ff40bd90-4116-4569-ac57-780ffeee857a nodeName:}" failed. No retries permitted until 2026-11-02 15:30:01.694912834 +0000 UTC m=+1903.897790872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-maas-api-tls") pod "maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" (UID: "ff40bd90-4116-4569-ac57-780ffeee857a") : secret "maas-api-e2e-shared-b-w2-ff6e1f-serving-cert" not found Nov 02 15:30:01.195127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.194956 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff40bd90-4116-4569-ac57-780ffeee857a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.195651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.195625 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff40bd90-4116-4569-ac57-780ffeee857a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.208974 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.208943 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl"] Nov 02 15:30:01.261586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.261544 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv"] Nov 02 15:30:01.272344 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.272317 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.272570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.272551 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m98z4\" (UniqueName: \"kubernetes.io/projected/ff40bd90-4116-4569-ac57-780ffeee857a-kube-api-access-m98z4\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.274139 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:30:01.274112 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe99deda_eff2_4a81_a338_8a683c54d176.slice/crio-1e89f0e40494098bf611f85a8b6c0a3edf484426da1945a1792040ce3f6ef39d WatchSource:0}: Error finding container 1e89f0e40494098bf611f85a8b6c0a3edf484426da1945a1792040ce3f6ef39d: Status 404 returned error can't find the container with id 1e89f0e40494098bf611f85a8b6c0a3edf484426da1945a1792040ce3f6ef39d Nov 02 15:30:01.274808 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:30:01.274750 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf577da_6f95_4a29_b4e3_d373978edcc9.slice/crio-c739fe5b6606e1addfd6ca5f3e3da29999e1debd2b6239f4174bfcc33ffb2fc4 WatchSource:0}: Error finding container c739fe5b6606e1addfd6ca5f3e3da29999e1debd2b6239f4174bfcc33ffb2fc4: Status 404 returned error can't find the container with id c739fe5b6606e1addfd6ca5f3e3da29999e1debd2b6239f4174bfcc33ffb2fc4 Nov 02 15:30:01.342094 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:30:01.342066 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead73082_e749_40ec_b4a5_e9583125a9aa.slice/crio-d30caf560676eeb10c4bd7761a3a5e76ed20da4a9e4fe9aad422f9c987ba84e8 WatchSource:0}: Error finding container d30caf560676eeb10c4bd7761a3a5e76ed20da4a9e4fe9aad422f9c987ba84e8: Status 404 returned error can't find the container with id d30caf560676eeb10c4bd7761a3a5e76ed20da4a9e4fe9aad422f9c987ba84e8 Nov 02 15:30:01.343084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.343042 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq"] Nov 02 15:30:01.351761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.350549 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5"] Nov 02 15:30:01.356097 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.356072 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs"] Nov 02 15:30:01.358644 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:30:01.358620 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d5e7e0_4be2_4f59_b268_df73ed648efe.slice/crio-92f37e72717bf331bc216a4f05e80b9d703f84ae5deb4860d5a836122ef4a771 WatchSource:0}: Error finding container 92f37e72717bf331bc216a4f05e80b9d703f84ae5deb4860d5a836122ef4a771: Status 404 returned error can't find the container with id 92f37e72717bf331bc216a4f05e80b9d703f84ae5deb4860d5a836122ef4a771 Nov 02 15:30:01.714396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.714354 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:01.717388 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:01.717359 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:02.012548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.012520 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:02.102750 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.102710 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" event={"ID":"13d5e7e0-4be2-4f59-b268-df73ed648efe","Type":"ContainerStarted","Data":"00d4614b179a53d7e7796ba8b7f465e74dbca018d2e63deceb63873904da1b46"} Nov 02 15:30:02.102912 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.102786 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" event={"ID":"13d5e7e0-4be2-4f59-b268-df73ed648efe","Type":"ContainerStarted","Data":"92f37e72717bf331bc216a4f05e80b9d703f84ae5deb4860d5a836122ef4a771"} Nov 02 15:30:02.105827 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.105792 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" event={"ID":"2c481a40-2914-4897-9b11-bb147f3cfadf","Type":"ContainerStarted","Data":"e1a9e05646840c5aa7101d433961a001e25ee112eb27f9d7b80cf0563d0156bf"} Nov 02 15:30:02.108707 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.108676 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" event={"ID":"3573e422-da50-42c9-98b9-3f544c6414e2","Type":"ContainerStarted","Data":"b2129c7db9c179a907e037a518c2c1df9cfa5afdda193dda1c442e368f30a1d7"} Nov 02 15:30:02.108895 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.108872 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" event={"ID":"3573e422-da50-42c9-98b9-3f544c6414e2","Type":"ContainerStarted","Data":"5b1138765c940d1f9b335440b1d3a29a175100ad68f4ad8497a3df469d2f7c2e"} Nov 02 15:30:02.110674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.110649 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" event={"ID":"ead73082-e749-40ec-b4a5-e9583125a9aa","Type":"ContainerStarted","Data":"92947a035930e6ae199cc5dbd19def13ef7edbacf521615a88a04f88eb1152ed"} Nov 02 15:30:02.110773 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.110697 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" event={"ID":"ead73082-e749-40ec-b4a5-e9583125a9aa","Type":"ContainerStarted","Data":"d30caf560676eeb10c4bd7761a3a5e76ed20da4a9e4fe9aad422f9c987ba84e8"} Nov 02 15:30:02.113192 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.113167 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" event={"ID":"bdf577da-6f95-4a29-b4e3-d373978edcc9","Type":"ContainerStarted","Data":"f82ab528d96809f69911330858656fb5d62b3d1008cd82884be7dcf71a370358"} Nov 02 15:30:02.113297 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.113212 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" event={"ID":"bdf577da-6f95-4a29-b4e3-d373978edcc9","Type":"ContainerStarted","Data":"c739fe5b6606e1addfd6ca5f3e3da29999e1debd2b6239f4174bfcc33ffb2fc4"} Nov 02 15:30:02.115030 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.115011 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" event={"ID":"be99deda-eff2-4a81-a338-8a683c54d176","Type":"ContainerStarted","Data":"1f9645dc10fdc75ed2ba334f5826cde3f097ae37dfca1632e9b11997a5cb7401"} Nov 02 15:30:02.115122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.115050 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" event={"ID":"be99deda-eff2-4a81-a338-8a683c54d176","Type":"ContainerStarted","Data":"1e89f0e40494098bf611f85a8b6c0a3edf484426da1945a1792040ce3f6ef39d"} Nov 02 15:30:02.218989 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.218940 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" podStartSLOduration=2.218927659 podStartE2EDuration="2.218927659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.216863652 +0000 UTC m=+1904.419741712" watchObservedRunningTime="2026-11-02 15:30:02.218927659 +0000 UTC m=+1904.421805718" Nov 02 15:30:02.335855 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.335806 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" podStartSLOduration=2.3357947550000002 podStartE2EDuration="2.335794755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.330227938 +0000 UTC m=+1904.533105998" watchObservedRunningTime="2026-11-02 15:30:02.335794755 +0000 UTC m=+1904.538672815" Nov 02 15:30:02.444465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.444405 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" podStartSLOduration=2.444389359 podStartE2EDuration="2.444389359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.440213018 +0000 UTC m=+1904.643091079" watchObservedRunningTime="2026-11-02 15:30:02.444389359 +0000 UTC m=+1904.647267418" Nov 02 15:30:02.509376 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.509347 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4"] Nov 02 15:30:02.565575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.565470 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" podStartSLOduration=2.5654557650000003 podStartE2EDuration="2.565455765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.561978663 +0000 UTC m=+1904.764856722" watchObservedRunningTime="2026-11-02 15:30:02.565455765 +0000 UTC m=+1904.768333824" Nov 02 15:30:02.588866 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:30:02.588831 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff40bd90_4116_4569_ac57_780ffeee857a.slice/crio-6511a267ed46978ff8c23525aedaca0ab74c0b0c1fd6a58a7418088ef543cb5c WatchSource:0}: Error finding container 6511a267ed46978ff8c23525aedaca0ab74c0b0c1fd6a58a7418088ef543cb5c: Status 404 returned error can't find the container with id 6511a267ed46978ff8c23525aedaca0ab74c0b0c1fd6a58a7418088ef543cb5c Nov 02 15:30:02.590536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.590516 5476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:30:02.675381 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.675317 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" podStartSLOduration=2.675298716 podStartE2EDuration="2.675298716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.671702327 +0000 UTC m=+1904.874580386" watchObservedRunningTime="2026-11-02 15:30:02.675298716 +0000 UTC m=+1904.878176778" Nov 02 15:30:02.783535 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:02.783478 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" podStartSLOduration=2.783464933 podStartE2EDuration="2.783464933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.781922235 +0000 UTC m=+1904.984800296" watchObservedRunningTime="2026-11-02 15:30:02.783464933 +0000 UTC m=+1904.986342992" Nov 02 15:30:03.123592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.123551 5476 generic.go:356] "Generic (PLEG): container finished" podID="3573e422-da50-42c9-98b9-3f544c6414e2" containerID="b2129c7db9c179a907e037a518c2c1df9cfa5afdda193dda1c442e368f30a1d7" exitCode=60 Nov 02 15:30:03.123856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.123639 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" event={"ID":"3573e422-da50-42c9-98b9-3f544c6414e2","Type":"ContainerDied","Data":"b2129c7db9c179a907e037a518c2c1df9cfa5afdda193dda1c442e368f30a1d7"} Nov 02 15:30:03.123968 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.123950 5476 scope.go:122] "RemoveContainer" containerID="b2129c7db9c179a907e037a518c2c1df9cfa5afdda193dda1c442e368f30a1d7" Nov 02 15:30:03.125490 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.125469 5476 generic.go:356] "Generic (PLEG): container finished" podID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerID="92947a035930e6ae199cc5dbd19def13ef7edbacf521615a88a04f88eb1152ed" exitCode=60 Nov 02 15:30:03.125608 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.125549 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" event={"ID":"ead73082-e749-40ec-b4a5-e9583125a9aa","Type":"ContainerDied","Data":"92947a035930e6ae199cc5dbd19def13ef7edbacf521615a88a04f88eb1152ed"} Nov 02 15:30:03.125849 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.125829 5476 scope.go:122] "RemoveContainer" containerID="92947a035930e6ae199cc5dbd19def13ef7edbacf521615a88a04f88eb1152ed" Nov 02 15:30:03.127373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.127316 5476 generic.go:356] "Generic (PLEG): container finished" podID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerID="f82ab528d96809f69911330858656fb5d62b3d1008cd82884be7dcf71a370358" exitCode=60 Nov 02 15:30:03.127447 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.127407 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" event={"ID":"bdf577da-6f95-4a29-b4e3-d373978edcc9","Type":"ContainerDied","Data":"f82ab528d96809f69911330858656fb5d62b3d1008cd82884be7dcf71a370358"} Nov 02 15:30:03.127692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.127610 5476 scope.go:122] "RemoveContainer" containerID="f82ab528d96809f69911330858656fb5d62b3d1008cd82884be7dcf71a370358" Nov 02 15:30:03.129152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.128917 5476 generic.go:356] "Generic (PLEG): container finished" podID="be99deda-eff2-4a81-a338-8a683c54d176" containerID="1f9645dc10fdc75ed2ba334f5826cde3f097ae37dfca1632e9b11997a5cb7401" exitCode=60 Nov 02 15:30:03.129152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.129021 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" event={"ID":"be99deda-eff2-4a81-a338-8a683c54d176","Type":"ContainerDied","Data":"1f9645dc10fdc75ed2ba334f5826cde3f097ae37dfca1632e9b11997a5cb7401"} Nov 02 15:30:03.129312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.129202 5476 scope.go:122] "RemoveContainer" containerID="1f9645dc10fdc75ed2ba334f5826cde3f097ae37dfca1632e9b11997a5cb7401" Nov 02 15:30:03.130341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.130316 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" event={"ID":"ff40bd90-4116-4569-ac57-780ffeee857a","Type":"ContainerStarted","Data":"6511a267ed46978ff8c23525aedaca0ab74c0b0c1fd6a58a7418088ef543cb5c"} Nov 02 15:30:03.131790 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.131769 5476 generic.go:356] "Generic (PLEG): container finished" podID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerID="00d4614b179a53d7e7796ba8b7f465e74dbca018d2e63deceb63873904da1b46" exitCode=60 Nov 02 15:30:03.131865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.131837 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" event={"ID":"13d5e7e0-4be2-4f59-b268-df73ed648efe","Type":"ContainerDied","Data":"00d4614b179a53d7e7796ba8b7f465e74dbca018d2e63deceb63873904da1b46"} Nov 02 15:30:03.132049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.132035 5476 scope.go:122] "RemoveContainer" containerID="00d4614b179a53d7e7796ba8b7f465e74dbca018d2e63deceb63873904da1b46" Nov 02 15:30:03.133438 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.133416 5476 generic.go:356] "Generic (PLEG): container finished" podID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerID="e1a9e05646840c5aa7101d433961a001e25ee112eb27f9d7b80cf0563d0156bf" exitCode=60 Nov 02 15:30:03.133528 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.133438 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" event={"ID":"2c481a40-2914-4897-9b11-bb147f3cfadf","Type":"ContainerDied","Data":"e1a9e05646840c5aa7101d433961a001e25ee112eb27f9d7b80cf0563d0156bf"} Nov 02 15:30:03.133675 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:03.133662 5476 scope.go:122] "RemoveContainer" containerID="e1a9e05646840c5aa7101d433961a001e25ee112eb27f9d7b80cf0563d0156bf" Nov 02 15:30:04.142949 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:04.142913 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" event={"ID":"13d5e7e0-4be2-4f59-b268-df73ed648efe","Type":"ContainerStarted","Data":"2b3785431f4efca16ed191adce7ff67330b25af0cd04f13eab05d25be136e4ac"} Nov 02 15:30:04.145052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:04.145020 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" event={"ID":"2c481a40-2914-4897-9b11-bb147f3cfadf","Type":"ContainerStarted","Data":"53756f5467cd0aca40a3d9b3cbdacbb7b6586f44c3e5287f42684e5aa6fef263"} Nov 02 15:30:04.146896 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:04.146873 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" event={"ID":"3573e422-da50-42c9-98b9-3f544c6414e2","Type":"ContainerStarted","Data":"5dc2c40eb3fe46b36a8f746c94f610d5cd0e47f9d44255287ab6628dc14ab9a7"} Nov 02 15:30:04.148836 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:04.148815 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" event={"ID":"ead73082-e749-40ec-b4a5-e9583125a9aa","Type":"ContainerStarted","Data":"d545e9ebac165fe3583b9e8a612b9c1642da66415f4a8e08c9af90ecd23c5a81"} Nov 02 15:30:04.150631 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:04.150607 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" event={"ID":"bdf577da-6f95-4a29-b4e3-d373978edcc9","Type":"ContainerStarted","Data":"7b4edf54ee24371cc2d56c00f69f99b4a0461934497404840db13f57ed92d870"} Nov 02 15:30:04.152416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:04.152393 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" event={"ID":"be99deda-eff2-4a81-a338-8a683c54d176","Type":"ContainerStarted","Data":"265c1cf6a4b535aaaf2fe434b5073a75b38bf7a0fd6e5225eca0892aa11d81a4"} Nov 02 15:30:04.153969 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:04.153948 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" event={"ID":"ff40bd90-4116-4569-ac57-780ffeee857a","Type":"ContainerStarted","Data":"27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d"} Nov 02 15:30:04.154115 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:04.154095 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:04.878543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:04.877462 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" podStartSLOduration=3.934461537 podStartE2EDuration="4.877441641s" totalImagesPullingTime="942.980104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:04.762218459 +0000 UTC m=+1906.965096524" watchObservedRunningTime="2026-11-02 15:30:04.877441641 +0000 UTC m=+1907.080319703" Nov 02 15:30:05.163804 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.163719 5476 generic.go:356] "Generic (PLEG): container finished" podID="be99deda-eff2-4a81-a338-8a683c54d176" containerID="265c1cf6a4b535aaaf2fe434b5073a75b38bf7a0fd6e5225eca0892aa11d81a4" exitCode=60 Nov 02 15:30:05.164261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.163807 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" event={"ID":"be99deda-eff2-4a81-a338-8a683c54d176","Type":"ContainerDied","Data":"265c1cf6a4b535aaaf2fe434b5073a75b38bf7a0fd6e5225eca0892aa11d81a4"} Nov 02 15:30:05.164261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.163863 5476 scope.go:122] "RemoveContainer" containerID="1f9645dc10fdc75ed2ba334f5826cde3f097ae37dfca1632e9b11997a5cb7401" Nov 02 15:30:05.164261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.164163 5476 scope.go:122] "RemoveContainer" containerID="265c1cf6a4b535aaaf2fe434b5073a75b38bf7a0fd6e5225eca0892aa11d81a4" Nov 02 15:30:05.164594 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:05.164567 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl_odh-ai-gateway-infra(be99deda-eff2-4a81-a338-8a683c54d176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" podUID="be99deda-eff2-4a81-a338-8a683c54d176" Nov 02 15:30:05.166036 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.165985 5476 generic.go:356] "Generic (PLEG): container finished" podID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerID="2b3785431f4efca16ed191adce7ff67330b25af0cd04f13eab05d25be136e4ac" exitCode=60 Nov 02 15:30:05.166143 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.166038 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" event={"ID":"13d5e7e0-4be2-4f59-b268-df73ed648efe","Type":"ContainerDied","Data":"2b3785431f4efca16ed191adce7ff67330b25af0cd04f13eab05d25be136e4ac"} Nov 02 15:30:05.166338 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.166320 5476 scope.go:122] "RemoveContainer" containerID="2b3785431f4efca16ed191adce7ff67330b25af0cd04f13eab05d25be136e4ac" Nov 02 15:30:05.166679 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:05.166660 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs_odh-ai-gateway-infra(13d5e7e0-4be2-4f59-b268-df73ed648efe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" Nov 02 15:30:05.167879 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.167860 5476 generic.go:356] "Generic (PLEG): container finished" podID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerID="53756f5467cd0aca40a3d9b3cbdacbb7b6586f44c3e5287f42684e5aa6fef263" exitCode=60 Nov 02 15:30:05.167965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.167943 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" event={"ID":"2c481a40-2914-4897-9b11-bb147f3cfadf","Type":"ContainerDied","Data":"53756f5467cd0aca40a3d9b3cbdacbb7b6586f44c3e5287f42684e5aa6fef263"} Nov 02 15:30:05.168239 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.168217 5476 scope.go:122] "RemoveContainer" containerID="53756f5467cd0aca40a3d9b3cbdacbb7b6586f44c3e5287f42684e5aa6fef263" Nov 02 15:30:05.168535 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:05.168516 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7_odh-ai-gateway-infra(2c481a40-2914-4897-9b11-bb147f3cfadf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" Nov 02 15:30:05.169785 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.169765 5476 generic.go:356] "Generic (PLEG): container finished" podID="3573e422-da50-42c9-98b9-3f544c6414e2" containerID="5dc2c40eb3fe46b36a8f746c94f610d5cd0e47f9d44255287ab6628dc14ab9a7" exitCode=60 Nov 02 15:30:05.169930 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.169827 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" event={"ID":"3573e422-da50-42c9-98b9-3f544c6414e2","Type":"ContainerDied","Data":"5dc2c40eb3fe46b36a8f746c94f610d5cd0e47f9d44255287ab6628dc14ab9a7"} Nov 02 15:30:05.170127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.170107 5476 scope.go:122] "RemoveContainer" containerID="5dc2c40eb3fe46b36a8f746c94f610d5cd0e47f9d44255287ab6628dc14ab9a7" Nov 02 15:30:05.170398 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:05.170380 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5_odh-ai-gateway-infra(3573e422-da50-42c9-98b9-3f544c6414e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" Nov 02 15:30:05.171812 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.171793 5476 generic.go:356] "Generic (PLEG): container finished" podID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerID="d545e9ebac165fe3583b9e8a612b9c1642da66415f4a8e08c9af90ecd23c5a81" exitCode=60 Nov 02 15:30:05.171898 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.171869 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" event={"ID":"ead73082-e749-40ec-b4a5-e9583125a9aa","Type":"ContainerDied","Data":"d545e9ebac165fe3583b9e8a612b9c1642da66415f4a8e08c9af90ecd23c5a81"} Nov 02 15:30:05.172140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.172125 5476 scope.go:122] "RemoveContainer" containerID="d545e9ebac165fe3583b9e8a612b9c1642da66415f4a8e08c9af90ecd23c5a81" Nov 02 15:30:05.172371 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:05.172342 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq_odh-ai-gateway-infra(ead73082-e749-40ec-b4a5-e9583125a9aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" Nov 02 15:30:05.173768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.173749 5476 generic.go:356] "Generic (PLEG): container finished" podID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerID="7b4edf54ee24371cc2d56c00f69f99b4a0461934497404840db13f57ed92d870" exitCode=60 Nov 02 15:30:05.173867 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.173829 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" event={"ID":"bdf577da-6f95-4a29-b4e3-d373978edcc9","Type":"ContainerDied","Data":"7b4edf54ee24371cc2d56c00f69f99b4a0461934497404840db13f57ed92d870"} Nov 02 15:30:05.174054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.174039 5476 scope.go:122] "RemoveContainer" containerID="7b4edf54ee24371cc2d56c00f69f99b4a0461934497404840db13f57ed92d870" Nov 02 15:30:05.174264 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:05.174245 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv_odh-ai-gateway-infra(bdf577da-6f95-4a29-b4e3-d373978edcc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" Nov 02 15:30:05.178077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.178060 5476 scope.go:122] "RemoveContainer" containerID="00d4614b179a53d7e7796ba8b7f465e74dbca018d2e63deceb63873904da1b46" Nov 02 15:30:05.192300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.192274 5476 scope.go:122] "RemoveContainer" containerID="e1a9e05646840c5aa7101d433961a001e25ee112eb27f9d7b80cf0563d0156bf" Nov 02 15:30:05.205566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.205549 5476 scope.go:122] "RemoveContainer" containerID="b2129c7db9c179a907e037a518c2c1df9cfa5afdda193dda1c442e368f30a1d7" Nov 02 15:30:05.218719 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.218699 5476 scope.go:122] "RemoveContainer" containerID="92947a035930e6ae199cc5dbd19def13ef7edbacf521615a88a04f88eb1152ed" Nov 02 15:30:05.232053 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:05.232034 5476 scope.go:122] "RemoveContainer" containerID="f82ab528d96809f69911330858656fb5d62b3d1008cd82884be7dcf71a370358" Nov 02 15:30:06.188283 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:06.188248 5476 scope.go:122] "RemoveContainer" containerID="d545e9ebac165fe3583b9e8a612b9c1642da66415f4a8e08c9af90ecd23c5a81" Nov 02 15:30:06.188942 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:06.188818 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq_odh-ai-gateway-infra(ead73082-e749-40ec-b4a5-e9583125a9aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" Nov 02 15:30:06.190731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:06.190710 5476 scope.go:122] "RemoveContainer" containerID="7b4edf54ee24371cc2d56c00f69f99b4a0461934497404840db13f57ed92d870" Nov 02 15:30:06.191049 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:06.191027 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv_odh-ai-gateway-infra(bdf577da-6f95-4a29-b4e3-d373978edcc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" Nov 02 15:30:06.192403 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:06.192387 5476 scope.go:122] "RemoveContainer" containerID="265c1cf6a4b535aaaf2fe434b5073a75b38bf7a0fd6e5225eca0892aa11d81a4" Nov 02 15:30:06.192601 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:06.192574 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl_odh-ai-gateway-infra(be99deda-eff2-4a81-a338-8a683c54d176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" podUID="be99deda-eff2-4a81-a338-8a683c54d176" Nov 02 15:30:06.194113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:06.194094 5476 scope.go:122] "RemoveContainer" containerID="2b3785431f4efca16ed191adce7ff67330b25af0cd04f13eab05d25be136e4ac" Nov 02 15:30:06.194314 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:06.194290 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs_odh-ai-gateway-infra(13d5e7e0-4be2-4f59-b268-df73ed648efe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" Nov 02 15:30:06.195738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:06.195715 5476 scope.go:122] "RemoveContainer" containerID="53756f5467cd0aca40a3d9b3cbdacbb7b6586f44c3e5287f42684e5aa6fef263" Nov 02 15:30:06.196060 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:06.196032 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7_odh-ai-gateway-infra(2c481a40-2914-4897-9b11-bb147f3cfadf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" Nov 02 15:30:06.197058 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:06.197043 5476 scope.go:122] "RemoveContainer" containerID="5dc2c40eb3fe46b36a8f746c94f610d5cd0e47f9d44255287ab6628dc14ab9a7" Nov 02 15:30:06.197225 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:06.197210 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5_odh-ai-gateway-infra(3573e422-da50-42c9-98b9-3f544c6414e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" Nov 02 15:30:10.180195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:10.180167 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:30:14.915531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:14.915499 5476 scope.go:122] "RemoveContainer" containerID="53756f5467cd0aca40a3d9b3cbdacbb7b6586f44c3e5287f42684e5aa6fef263" Nov 02 15:30:16.283649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:16.283563 5476 generic.go:356] "Generic (PLEG): container finished" podID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerID="069efa920d66250cf5bb6904dcef0c6adb36958ac08df7a11132eb5e4c1ecbcf" exitCode=60 Nov 02 15:30:16.284087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:16.283641 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" event={"ID":"2c481a40-2914-4897-9b11-bb147f3cfadf","Type":"ContainerDied","Data":"069efa920d66250cf5bb6904dcef0c6adb36958ac08df7a11132eb5e4c1ecbcf"} Nov 02 15:30:16.284087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:16.283710 5476 scope.go:122] "RemoveContainer" containerID="53756f5467cd0aca40a3d9b3cbdacbb7b6586f44c3e5287f42684e5aa6fef263" Nov 02 15:30:16.284199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:16.284185 5476 scope.go:122] "RemoveContainer" containerID="069efa920d66250cf5bb6904dcef0c6adb36958ac08df7a11132eb5e4c1ecbcf" Nov 02 15:30:16.284585 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:16.284562 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7_odh-ai-gateway-infra(2c481a40-2914-4897-9b11-bb147f3cfadf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" Nov 02 15:30:16.915520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:16.915491 5476 scope.go:122] "RemoveContainer" containerID="5dc2c40eb3fe46b36a8f746c94f610d5cd0e47f9d44255287ab6628dc14ab9a7" Nov 02 15:30:17.294781 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:17.294749 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" event={"ID":"3573e422-da50-42c9-98b9-3f544c6414e2","Type":"ContainerStarted","Data":"552a60d3c879a41eb5975cf168d23b5ef1f8696eb7326767bfa48a58970b2877"} Nov 02 15:30:17.404519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:17.404478 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7"] Nov 02 15:30:17.591194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:17.591167 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" Nov 02 15:30:17.760915 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:17.760821 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c481a40-2914-4897-9b11-bb147f3cfadf-kube-api-access-qn48q\" (UniqueName: \"kubernetes.io/projected/2c481a40-2914-4897-9b11-bb147f3cfadf-kube-api-access-qn48q\") pod \"2c481a40-2914-4897-9b11-bb147f3cfadf\" (UID: \"2c481a40-2914-4897-9b11-bb147f3cfadf\") " Nov 02 15:30:17.761140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:17.760928 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c481a40-2914-4897-9b11-bb147f3cfadf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c481a40-2914-4897-9b11-bb147f3cfadf-maas-api-service-ca\") pod \"2c481a40-2914-4897-9b11-bb147f3cfadf\" (UID: \"2c481a40-2914-4897-9b11-bb147f3cfadf\") " Nov 02 15:30:17.762500 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:17.762468 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c481a40-2914-4897-9b11-bb147f3cfadf-maas-api-service-ca" pod "2c481a40-2914-4897-9b11-bb147f3cfadf" (UID: "2c481a40-2914-4897-9b11-bb147f3cfadf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:17.771810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:17.771638 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c481a40-2914-4897-9b11-bb147f3cfadf-kube-api-access-qn48q" pod "2c481a40-2914-4897-9b11-bb147f3cfadf" (UID: "2c481a40-2914-4897-9b11-bb147f3cfadf"). InnerVolumeSpecName "kube-api-access-qn48q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:17.863368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:17.863334 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn48q\" (UniqueName: \"kubernetes.io/projected/2c481a40-2914-4897-9b11-bb147f3cfadf-kube-api-access-qn48q\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:17.863368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:17.863363 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c481a40-2914-4897-9b11-bb147f3cfadf-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:17.920612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:17.920562 5476 scope.go:122] "RemoveContainer" containerID="7b4edf54ee24371cc2d56c00f69f99b4a0461934497404840db13f57ed92d870" Nov 02 15:30:18.304372 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.304332 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" event={"ID":"bdf577da-6f95-4a29-b4e3-d373978edcc9","Type":"ContainerStarted","Data":"3c4f2264b0375fe7cd755bcba795888ce4467890ca963a36aaa516e27919f7cc"} Nov 02 15:30:18.305960 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.305940 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" Nov 02 15:30:18.306073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.305980 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7" event={"ID":"2c481a40-2914-4897-9b11-bb147f3cfadf","Type":"ContainerDied","Data":"1e29bfefa9acd43566ca25f9b16b8fee1ad97d6da12580e18da30d59cd401afd"} Nov 02 15:30:18.306073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.306049 5476 scope.go:122] "RemoveContainer" containerID="069efa920d66250cf5bb6904dcef0c6adb36958ac08df7a11132eb5e4c1ecbcf" Nov 02 15:30:18.308195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.308171 5476 generic.go:356] "Generic (PLEG): container finished" podID="3573e422-da50-42c9-98b9-3f544c6414e2" containerID="552a60d3c879a41eb5975cf168d23b5ef1f8696eb7326767bfa48a58970b2877" exitCode=60 Nov 02 15:30:18.308319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.308228 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" event={"ID":"3573e422-da50-42c9-98b9-3f544c6414e2","Type":"ContainerDied","Data":"552a60d3c879a41eb5975cf168d23b5ef1f8696eb7326767bfa48a58970b2877"} Nov 02 15:30:18.308655 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.308635 5476 scope.go:122] "RemoveContainer" containerID="552a60d3c879a41eb5975cf168d23b5ef1f8696eb7326767bfa48a58970b2877" Nov 02 15:30:18.308925 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:18.308906 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5_odh-ai-gateway-infra(3573e422-da50-42c9-98b9-3f544c6414e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" Nov 02 15:30:18.316850 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.316829 5476 scope.go:122] "RemoveContainer" containerID="5dc2c40eb3fe46b36a8f746c94f610d5cd0e47f9d44255287ab6628dc14ab9a7" Nov 02 15:30:18.423848 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.423809 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5"] Nov 02 15:30:18.519471 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.519431 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7"] Nov 02 15:30:18.524830 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.524793 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893890-v86w7"] Nov 02 15:30:18.915331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.915306 5476 scope.go:122] "RemoveContainer" containerID="d545e9ebac165fe3583b9e8a612b9c1642da66415f4a8e08c9af90ecd23c5a81" Nov 02 15:30:18.915525 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.915505 5476 scope.go:122] "RemoveContainer" containerID="265c1cf6a4b535aaaf2fe434b5073a75b38bf7a0fd6e5225eca0892aa11d81a4" Nov 02 15:30:18.915668 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:18.915649 5476 scope.go:122] "RemoveContainer" containerID="2b3785431f4efca16ed191adce7ff67330b25af0cd04f13eab05d25be136e4ac" Nov 02 15:30:19.317509 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.317480 5476 generic.go:356] "Generic (PLEG): container finished" podID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerID="3c4f2264b0375fe7cd755bcba795888ce4467890ca963a36aaa516e27919f7cc" exitCode=60 Nov 02 15:30:19.317876 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.317558 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" event={"ID":"bdf577da-6f95-4a29-b4e3-d373978edcc9","Type":"ContainerDied","Data":"3c4f2264b0375fe7cd755bcba795888ce4467890ca963a36aaa516e27919f7cc"} Nov 02 15:30:19.317876 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.317645 5476 scope.go:122] "RemoveContainer" containerID="7b4edf54ee24371cc2d56c00f69f99b4a0461934497404840db13f57ed92d870" Nov 02 15:30:19.318072 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.318054 5476 scope.go:122] "RemoveContainer" containerID="3c4f2264b0375fe7cd755bcba795888ce4467890ca963a36aaa516e27919f7cc" Nov 02 15:30:19.318408 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:19.318381 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv_odh-ai-gateway-infra(bdf577da-6f95-4a29-b4e3-d373978edcc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" Nov 02 15:30:19.425481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.425452 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv"] Nov 02 15:30:19.487859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.487830 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" Nov 02 15:30:19.619017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.614738 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3573e422-da50-42c9-98b9-3f544c6414e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3573e422-da50-42c9-98b9-3f544c6414e2-maas-api-service-ca\") pod \"3573e422-da50-42c9-98b9-3f544c6414e2\" (UID: \"3573e422-da50-42c9-98b9-3f544c6414e2\") " Nov 02 15:30:19.619017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.615108 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3573e422-da50-42c9-98b9-3f544c6414e2-kube-api-access-kqb4j\" (UniqueName: \"kubernetes.io/projected/3573e422-da50-42c9-98b9-3f544c6414e2-kube-api-access-kqb4j\") pod \"3573e422-da50-42c9-98b9-3f544c6414e2\" (UID: \"3573e422-da50-42c9-98b9-3f544c6414e2\") " Nov 02 15:30:19.640017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.636879 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3573e422-da50-42c9-98b9-3f544c6414e2-maas-api-service-ca" pod "3573e422-da50-42c9-98b9-3f544c6414e2" (UID: "3573e422-da50-42c9-98b9-3f544c6414e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:19.640017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.637685 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3573e422-da50-42c9-98b9-3f544c6414e2-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:19.653102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.650131 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3573e422-da50-42c9-98b9-3f544c6414e2-kube-api-access-kqb4j" pod "3573e422-da50-42c9-98b9-3f544c6414e2" (UID: "3573e422-da50-42c9-98b9-3f544c6414e2"). InnerVolumeSpecName "kube-api-access-kqb4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:19.739984 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.739377 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqb4j\" (UniqueName: \"kubernetes.io/projected/3573e422-da50-42c9-98b9-3f544c6414e2-kube-api-access-kqb4j\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:19.872141 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.872055 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:30:19.923813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:19.923778 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" path="/var/lib/kubelet/pods/2c481a40-2914-4897-9b11-bb147f3cfadf/volumes" Nov 02 15:30:20.149252 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:20.149161 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:30:20.343021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.338735 5476 generic.go:356] "Generic (PLEG): container finished" podID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerID="8cf0b886c96ab420934de2658c0a9105e0779204c9fefabb11d88f74d32d09ef" exitCode=60 Nov 02 15:30:20.343021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.338871 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" event={"ID":"13d5e7e0-4be2-4f59-b268-df73ed648efe","Type":"ContainerDied","Data":"8cf0b886c96ab420934de2658c0a9105e0779204c9fefabb11d88f74d32d09ef"} Nov 02 15:30:20.343021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.338959 5476 scope.go:122] "RemoveContainer" containerID="2b3785431f4efca16ed191adce7ff67330b25af0cd04f13eab05d25be136e4ac" Nov 02 15:30:20.343021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.339517 5476 scope.go:122] "RemoveContainer" containerID="8cf0b886c96ab420934de2658c0a9105e0779204c9fefabb11d88f74d32d09ef" Nov 02 15:30:20.343021 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:20.339921 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs_odh-ai-gateway-infra(13d5e7e0-4be2-4f59-b268-df73ed648efe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" Nov 02 15:30:20.346311 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.344828 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" Nov 02 15:30:20.346311 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.345727 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5" event={"ID":"3573e422-da50-42c9-98b9-3f544c6414e2","Type":"ContainerDied","Data":"5b1138765c940d1f9b335440b1d3a29a175100ad68f4ad8497a3df469d2f7c2e"} Nov 02 15:30:20.349746 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.348587 5476 generic.go:356] "Generic (PLEG): container finished" podID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerID="47b8c98e0c73a8f6d23befa76bfbb354e35f2e47aade0507eb12a2a2b8825dbd" exitCode=60 Nov 02 15:30:20.349746 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.348707 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" event={"ID":"ead73082-e749-40ec-b4a5-e9583125a9aa","Type":"ContainerDied","Data":"47b8c98e0c73a8f6d23befa76bfbb354e35f2e47aade0507eb12a2a2b8825dbd"} Nov 02 15:30:20.349746 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.349172 5476 scope.go:122] "RemoveContainer" containerID="47b8c98e0c73a8f6d23befa76bfbb354e35f2e47aade0507eb12a2a2b8825dbd" Nov 02 15:30:20.349746 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:20.349535 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq_odh-ai-gateway-infra(ead73082-e749-40ec-b4a5-e9583125a9aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" Nov 02 15:30:20.368091 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.359961 5476 generic.go:356] "Generic (PLEG): container finished" podID="be99deda-eff2-4a81-a338-8a683c54d176" containerID="10c65fb8717ac49d889e0ef3186a38935ba76bbff4a08def9c16dab5582bb80c" exitCode=60 Nov 02 15:30:20.368091 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.360111 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" event={"ID":"be99deda-eff2-4a81-a338-8a683c54d176","Type":"ContainerDied","Data":"10c65fb8717ac49d889e0ef3186a38935ba76bbff4a08def9c16dab5582bb80c"} Nov 02 15:30:20.368091 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.360612 5476 scope.go:122] "RemoveContainer" containerID="10c65fb8717ac49d889e0ef3186a38935ba76bbff4a08def9c16dab5582bb80c" Nov 02 15:30:20.368091 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:20.361062 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl_odh-ai-gateway-infra(be99deda-eff2-4a81-a338-8a683c54d176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" podUID="be99deda-eff2-4a81-a338-8a683c54d176" Nov 02 15:30:20.368830 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.368794 5476 scope.go:122] "RemoveContainer" containerID="552a60d3c879a41eb5975cf168d23b5ef1f8696eb7326767bfa48a58970b2877" Nov 02 15:30:20.394469 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.394444 5476 scope.go:122] "RemoveContainer" containerID="d545e9ebac165fe3583b9e8a612b9c1642da66415f4a8e08c9af90ecd23c5a81" Nov 02 15:30:20.425582 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.424983 5476 scope.go:122] "RemoveContainer" containerID="265c1cf6a4b535aaaf2fe434b5073a75b38bf7a0fd6e5225eca0892aa11d81a4" Nov 02 15:30:20.762921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.762877 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" Nov 02 15:30:20.823018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.822082 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5"] Nov 02 15:30:20.825487 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.825459 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893890-lnnx5"] Nov 02 15:30:20.893126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.892659 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf577da-6f95-4a29-b4e3-d373978edcc9-kube-api-access-j8kq4\" (UniqueName: \"kubernetes.io/projected/bdf577da-6f95-4a29-b4e3-d373978edcc9-kube-api-access-j8kq4\") pod \"bdf577da-6f95-4a29-b4e3-d373978edcc9\" (UID: \"bdf577da-6f95-4a29-b4e3-d373978edcc9\") " Nov 02 15:30:20.893126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.892741 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf577da-6f95-4a29-b4e3-d373978edcc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf577da-6f95-4a29-b4e3-d373978edcc9-maas-api-service-ca\") pod \"bdf577da-6f95-4a29-b4e3-d373978edcc9\" (UID: \"bdf577da-6f95-4a29-b4e3-d373978edcc9\") " Nov 02 15:30:20.899057 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:20.893509 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf577da-6f95-4a29-b4e3-d373978edcc9-maas-api-service-ca" pod "bdf577da-6f95-4a29-b4e3-d373978edcc9" (UID: "bdf577da-6f95-4a29-b4e3-d373978edcc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:21.045796 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.013279 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdf577da-6f95-4a29-b4e3-d373978edcc9-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.107200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.092287 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf577da-6f95-4a29-b4e3-d373978edcc9-kube-api-access-j8kq4" pod "bdf577da-6f95-4a29-b4e3-d373978edcc9" (UID: "bdf577da-6f95-4a29-b4e3-d373978edcc9"). InnerVolumeSpecName "kube-api-access-j8kq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:21.117988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.114709 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8kq4\" (UniqueName: \"kubernetes.io/projected/bdf577da-6f95-4a29-b4e3-d373978edcc9-kube-api-access-j8kq4\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.371772 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.371738 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" Nov 02 15:30:21.371772 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.371760 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv" event={"ID":"bdf577da-6f95-4a29-b4e3-d373978edcc9","Type":"ContainerDied","Data":"c739fe5b6606e1addfd6ca5f3e3da29999e1debd2b6239f4174bfcc33ffb2fc4"} Nov 02 15:30:21.372430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.371835 5476 scope.go:122] "RemoveContainer" containerID="3c4f2264b0375fe7cd755bcba795888ce4467890ca963a36aaa516e27919f7cc" Nov 02 15:30:21.465967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.465929 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs"] Nov 02 15:30:21.537603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.537534 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv"] Nov 02 15:30:21.544242 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.544212 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-941182-29893890-cjvjv"] Nov 02 15:30:21.625445 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.625381 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" Nov 02 15:30:21.723773 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.723730 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d5e7e0-4be2-4f59-b268-df73ed648efe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d5e7e0-4be2-4f59-b268-df73ed648efe-maas-api-service-ca\") pod \"13d5e7e0-4be2-4f59-b268-df73ed648efe\" (UID: \"13d5e7e0-4be2-4f59-b268-df73ed648efe\") " Nov 02 15:30:21.723987 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.723824 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d5e7e0-4be2-4f59-b268-df73ed648efe-kube-api-access-hrqgs\" (UniqueName: \"kubernetes.io/projected/13d5e7e0-4be2-4f59-b268-df73ed648efe-kube-api-access-hrqgs\") pod \"13d5e7e0-4be2-4f59-b268-df73ed648efe\" (UID: \"13d5e7e0-4be2-4f59-b268-df73ed648efe\") " Nov 02 15:30:21.724164 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.724140 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d5e7e0-4be2-4f59-b268-df73ed648efe-maas-api-service-ca" pod "13d5e7e0-4be2-4f59-b268-df73ed648efe" (UID: "13d5e7e0-4be2-4f59-b268-df73ed648efe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:21.724669 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.724648 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d5e7e0-4be2-4f59-b268-df73ed648efe-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.727105 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.727079 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl"] Nov 02 15:30:21.727230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.727150 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d5e7e0-4be2-4f59-b268-df73ed648efe-kube-api-access-hrqgs" pod "13d5e7e0-4be2-4f59-b268-df73ed648efe" (UID: "13d5e7e0-4be2-4f59-b268-df73ed648efe"). InnerVolumeSpecName "kube-api-access-hrqgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:21.826577 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.826537 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrqgs\" (UniqueName: \"kubernetes.io/projected/13d5e7e0-4be2-4f59-b268-df73ed648efe-kube-api-access-hrqgs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.892370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.892343 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" Nov 02 15:30:21.924680 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.924648 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" path="/var/lib/kubelet/pods/3573e422-da50-42c9-98b9-3f544c6414e2/volumes" Nov 02 15:30:21.925145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.925131 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" path="/var/lib/kubelet/pods/bdf577da-6f95-4a29-b4e3-d373978edcc9/volumes" Nov 02 15:30:21.943393 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:21.943357 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq"] Nov 02 15:30:22.031043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.028195 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be99deda-eff2-4a81-a338-8a683c54d176-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be99deda-eff2-4a81-a338-8a683c54d176-maas-api-service-ca\") pod \"be99deda-eff2-4a81-a338-8a683c54d176\" (UID: \"be99deda-eff2-4a81-a338-8a683c54d176\") " Nov 02 15:30:22.031043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.028242 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be99deda-eff2-4a81-a338-8a683c54d176-kube-api-access-hkjxv\" (UniqueName: \"kubernetes.io/projected/be99deda-eff2-4a81-a338-8a683c54d176-kube-api-access-hkjxv\") pod \"be99deda-eff2-4a81-a338-8a683c54d176\" (UID: \"be99deda-eff2-4a81-a338-8a683c54d176\") " Nov 02 15:30:22.031043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.028796 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be99deda-eff2-4a81-a338-8a683c54d176-maas-api-service-ca" pod "be99deda-eff2-4a81-a338-8a683c54d176" (UID: "be99deda-eff2-4a81-a338-8a683c54d176"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:22.031043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.029921 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be99deda-eff2-4a81-a338-8a683c54d176-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:22.031916 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.031875 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be99deda-eff2-4a81-a338-8a683c54d176-kube-api-access-hkjxv" pod "be99deda-eff2-4a81-a338-8a683c54d176" (UID: "be99deda-eff2-4a81-a338-8a683c54d176"). InnerVolumeSpecName "kube-api-access-hkjxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:22.116981 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.116954 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" Nov 02 15:30:22.131405 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.131340 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkjxv\" (UniqueName: \"kubernetes.io/projected/be99deda-eff2-4a81-a338-8a683c54d176-kube-api-access-hkjxv\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:22.232784 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.232751 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead73082-e749-40ec-b4a5-e9583125a9aa-kube-api-access-5brd9\" (UniqueName: \"kubernetes.io/projected/ead73082-e749-40ec-b4a5-e9583125a9aa-kube-api-access-5brd9\") pod \"ead73082-e749-40ec-b4a5-e9583125a9aa\" (UID: \"ead73082-e749-40ec-b4a5-e9583125a9aa\") " Nov 02 15:30:22.232970 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.232863 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead73082-e749-40ec-b4a5-e9583125a9aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead73082-e749-40ec-b4a5-e9583125a9aa-maas-api-service-ca\") pod \"ead73082-e749-40ec-b4a5-e9583125a9aa\" (UID: \"ead73082-e749-40ec-b4a5-e9583125a9aa\") " Nov 02 15:30:22.233345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.233313 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead73082-e749-40ec-b4a5-e9583125a9aa-maas-api-service-ca" pod "ead73082-e749-40ec-b4a5-e9583125a9aa" (UID: "ead73082-e749-40ec-b4a5-e9583125a9aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:22.233526 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.233502 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead73082-e749-40ec-b4a5-e9583125a9aa-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:22.235603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.235572 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead73082-e749-40ec-b4a5-e9583125a9aa-kube-api-access-5brd9" pod "ead73082-e749-40ec-b4a5-e9583125a9aa" (UID: "ead73082-e749-40ec-b4a5-e9583125a9aa"). InnerVolumeSpecName "kube-api-access-5brd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:22.335314 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.335276 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5brd9\" (UniqueName: \"kubernetes.io/projected/ead73082-e749-40ec-b4a5-e9583125a9aa-kube-api-access-5brd9\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:22.383448 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.383358 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" Nov 02 15:30:22.383448 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.383374 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs" event={"ID":"13d5e7e0-4be2-4f59-b268-df73ed648efe","Type":"ContainerDied","Data":"92f37e72717bf331bc216a4f05e80b9d703f84ae5deb4860d5a836122ef4a771"} Nov 02 15:30:22.383954 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.383476 5476 scope.go:122] "RemoveContainer" containerID="8cf0b886c96ab420934de2658c0a9105e0779204c9fefabb11d88f74d32d09ef" Nov 02 15:30:22.384976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.384932 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" Nov 02 15:30:22.385113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.384932 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq" event={"ID":"ead73082-e749-40ec-b4a5-e9583125a9aa","Type":"ContainerDied","Data":"d30caf560676eeb10c4bd7761a3a5e76ed20da4a9e4fe9aad422f9c987ba84e8"} Nov 02 15:30:22.387769 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.387746 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" event={"ID":"be99deda-eff2-4a81-a338-8a683c54d176","Type":"ContainerDied","Data":"1e89f0e40494098bf611f85a8b6c0a3edf484426da1945a1792040ce3f6ef39d"} Nov 02 15:30:22.387862 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.387791 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl" Nov 02 15:30:22.394114 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.394092 5476 scope.go:122] "RemoveContainer" containerID="47b8c98e0c73a8f6d23befa76bfbb354e35f2e47aade0507eb12a2a2b8825dbd" Nov 02 15:30:22.414331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.414298 5476 scope.go:122] "RemoveContainer" containerID="10c65fb8717ac49d889e0ef3186a38935ba76bbff4a08def9c16dab5582bb80c" Nov 02 15:30:22.540282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.540248 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl"] Nov 02 15:30:22.543967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.543936 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893890-lmpdl"] Nov 02 15:30:22.652301 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.652207 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs"] Nov 02 15:30:22.655732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.655704 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5aeac88e-29893890-mxqzs"] Nov 02 15:30:22.759335 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.759299 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq"] Nov 02 15:30:22.767203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:22.767173 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893890-cn8hq"] Nov 02 15:30:23.925603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:23.925570 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" path="/var/lib/kubelet/pods/13d5e7e0-4be2-4f59-b268-df73ed648efe/volumes" Nov 02 15:30:23.926116 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:23.926101 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be99deda-eff2-4a81-a338-8a683c54d176" path="/var/lib/kubelet/pods/be99deda-eff2-4a81-a338-8a683c54d176/volumes" Nov 02 15:30:23.926508 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:23.926496 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" path="/var/lib/kubelet/pods/ead73082-e749-40ec-b4a5-e9583125a9aa/volumes" Nov 02 15:30:43.723047 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.723007 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z"] Nov 02 15:30:43.723951 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.723932 5476 cpu_manager.go:414] "containerMap: removing container" podUID="be99deda-eff2-4a81-a338-8a683c54d176" containerName="cleanup" Nov 02 15:30:43.723951 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.723952 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be99deda-eff2-4a81-a338-8a683c54d176" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.723977 5476 cpu_manager.go:414] "containerMap: removing container" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.723985 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724008 5476 cpu_manager.go:414] "containerMap: removing container" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724017 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724033 5476 cpu_manager.go:414] "containerMap: removing container" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724040 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724051 5476 cpu_manager.go:414] "containerMap: removing container" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724058 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724078 5476 cpu_manager.go:414] "containerMap: removing container" podUID="be99deda-eff2-4a81-a338-8a683c54d176" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724084 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be99deda-eff2-4a81-a338-8a683c54d176" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724098 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724104 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724119 5476 cpu_manager.go:414] "containerMap: removing container" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" containerName="cleanup" Nov 02 15:30:43.724132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724126 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724162 5476 cpu_manager.go:414] "containerMap: removing container" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724171 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724182 5476 cpu_manager.go:414] "containerMap: removing container" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724189 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724202 5476 cpu_manager.go:414] "containerMap: removing container" podUID="be99deda-eff2-4a81-a338-8a683c54d176" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724209 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be99deda-eff2-4a81-a338-8a683c54d176" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724248 5476 cpu_manager.go:414] "containerMap: removing container" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724256 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724273 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724280 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724296 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724303 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724427 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724437 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724450 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724458 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="be99deda-eff2-4a81-a338-8a683c54d176" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724469 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724479 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724487 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724496 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724508 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="be99deda-eff2-4a81-a338-8a683c54d176" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724516 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724523 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724530 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerName="cleanup" Nov 02 15:30:43.724650 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.724540 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerName="cleanup" Nov 02 15:30:43.728520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.728503 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.777560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.777531 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-fb80ca-openshift-default-dockercfg-v2lf7\"" Nov 02 15:30:43.827542 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.827505 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z"] Nov 02 15:30:43.856521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.856490 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-data\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.856740 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.856534 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-credential-socket\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.856740 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.856650 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istiod-ca-cert\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.856872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.856774 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-envoy\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.856928 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.856908 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pk9n\" (UniqueName: \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-kube-api-access-6pk9n\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.857040 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.856972 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-podinfo\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.857150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.857132 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-certs\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.857212 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.857192 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-socket\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.857299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.857277 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-token\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.958834 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.958796 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-socket\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959060 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.958849 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-token\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959060 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.958918 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-data\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959060 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.958963 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-credential-socket\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.959084 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istiod-ca-cert\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.959149 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-envoy\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.959254 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pk9n\" (UniqueName: \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-kube-api-access-6pk9n\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.959292 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-podinfo\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.959348 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-data\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.959367 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-credential-socket\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.959390 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-certs\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959600 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.959569 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-socket\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.959727 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istiod-ca-cert\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.959817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.959725 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-certs\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.961556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.961535 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-envoy\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:43.961664 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:43.961639 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-podinfo\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:44.031708 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:44.031619 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pk9n\" (UniqueName: \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-kube-api-access-6pk9n\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:44.038551 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:44.038515 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-token\") pod \"e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:44.043143 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:44.043111 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:44.489717 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:44.489679 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z"] Nov 02 15:30:44.494664 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:30:44.494638 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8291c3e_4c32_43ef_a0d3_c4ed85b1eb87.slice/crio-aef577be96736ad49555033510a1aef17125f9c82e9fb6dd54e4af5200810cba WatchSource:0}: Error finding container aef577be96736ad49555033510a1aef17125f9c82e9fb6dd54e4af5200810cba: Status 404 returned error can't find the container with id aef577be96736ad49555033510a1aef17125f9c82e9fb6dd54e4af5200810cba Nov 02 15:30:44.497299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:44.497268 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:30:44.497368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:44.497339 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:30:44.497405 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:44.497369 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:30:44.575321 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:44.575280 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" event={"ID":"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87","Type":"ContainerStarted","Data":"f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878"} Nov 02 15:30:44.575595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:44.575574 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" event={"ID":"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87","Type":"ContainerStarted","Data":"aef577be96736ad49555033510a1aef17125f9c82e9fb6dd54e4af5200810cba"} Nov 02 15:30:44.694352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:44.694291 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" podStartSLOduration=1.6942735039999999 podStartE2EDuration="1.694273504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:44.690867539 +0000 UTC m=+1946.893745599" watchObservedRunningTime="2026-11-02 15:30:44.694273504 +0000 UTC m=+1946.897151565" Nov 02 15:30:45.043317 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:45.043267 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:45.050437 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:45.050409 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:45.582977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:45.582944 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:45.584015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:45.583974 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:30:50.147426 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:30:50.147387 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:30:51.384716 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.384647 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9"] Nov 02 15:30:51.386875 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.386846 5476 cpu_manager.go:414] "containerMap: removing container" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" containerName="cleanup" Nov 02 15:30:51.387049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.387035 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" containerName="cleanup" Nov 02 15:30:51.387175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.387164 5476 cpu_manager.go:414] "containerMap: removing container" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerName="cleanup" Nov 02 15:30:51.387256 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.387245 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerName="cleanup" Nov 02 15:30:51.387505 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.387490 5476 cpu_manager.go:414] "containerMap: removing container" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerName="cleanup" Nov 02 15:30:51.387602 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.387590 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerName="cleanup" Nov 02 15:30:51.387929 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.387913 5476 cpu_manager.go:414] "containerMap: removing container" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerName="cleanup" Nov 02 15:30:51.388111 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.388096 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf577da-6f95-4a29-b4e3-d373978edcc9" containerName="cleanup" Nov 02 15:30:51.388471 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.388454 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead73082-e749-40ec-b4a5-e9583125a9aa" containerName="cleanup" Nov 02 15:30:51.388580 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.388568 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d5e7e0-4be2-4f59-b268-df73ed648efe" containerName="cleanup" Nov 02 15:30:51.388678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.388668 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3573e422-da50-42c9-98b9-3f544c6414e2" containerName="cleanup" Nov 02 15:30:51.388758 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.388749 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="be99deda-eff2-4a81-a338-8a683c54d176" containerName="cleanup" Nov 02 15:30:51.388836 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.388827 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c481a40-2914-4897-9b11-bb147f3cfadf" containerName="cleanup" Nov 02 15:30:51.411015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.397384 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:30:51.499046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.498489 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9"] Nov 02 15:30:51.535019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.533375 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-fb80ca\"" Nov 02 15:30:51.535019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.533563 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-fb80ca-dockercfg-pkvhl\"" Nov 02 15:30:51.602244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.602214 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvpsg\" (UniqueName: \"kubernetes.io/projected/06a08d9f-a36e-4a36-90ef-414f5c1ae379-kube-api-access-cvpsg\") pod \"payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9\" (UID: \"06a08d9f-a36e-4a36-90ef-414f5c1ae379\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:30:51.602415 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.602333 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a08d9f-a36e-4a36-90ef-414f5c1ae379-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9\" (UID: \"06a08d9f-a36e-4a36-90ef-414f5c1ae379\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:30:51.703547 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.703464 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvpsg\" (UniqueName: \"kubernetes.io/projected/06a08d9f-a36e-4a36-90ef-414f5c1ae379-kube-api-access-cvpsg\") pod \"payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9\" (UID: \"06a08d9f-a36e-4a36-90ef-414f5c1ae379\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:30:51.703547 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.703532 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a08d9f-a36e-4a36-90ef-414f5c1ae379-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9\" (UID: \"06a08d9f-a36e-4a36-90ef-414f5c1ae379\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:30:51.704266 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.704245 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a08d9f-a36e-4a36-90ef-414f5c1ae379-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9\" (UID: \"06a08d9f-a36e-4a36-90ef-414f5c1ae379\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:30:51.761697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.761672 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvpsg\" (UniqueName: \"kubernetes.io/projected/06a08d9f-a36e-4a36-90ef-414f5c1ae379-kube-api-access-cvpsg\") pod \"payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9\" (UID: \"06a08d9f-a36e-4a36-90ef-414f5c1ae379\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:30:51.853770 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:51.853742 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:30:52.229672 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:52.229639 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9"] Nov 02 15:30:52.234110 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:30:52.234072 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a08d9f_a36e_4a36_90ef_414f5c1ae379.slice/crio-a8235727da04b1f9336e8012483b4242704ad99f68247a4411f5722a2fb83053 WatchSource:0}: Error finding container a8235727da04b1f9336e8012483b4242704ad99f68247a4411f5722a2fb83053: Status 404 returned error can't find the container with id a8235727da04b1f9336e8012483b4242704ad99f68247a4411f5722a2fb83053 Nov 02 15:30:52.648147 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:52.648094 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" event={"ID":"06a08d9f-a36e-4a36-90ef-414f5c1ae379","Type":"ContainerStarted","Data":"4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7"} Nov 02 15:30:52.648874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:52.648200 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" event={"ID":"06a08d9f-a36e-4a36-90ef-414f5c1ae379","Type":"ContainerStarted","Data":"a8235727da04b1f9336e8012483b4242704ad99f68247a4411f5722a2fb83053"} Nov 02 15:30:52.648874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:52.648245 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:30:52.759837 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:30:52.759785 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" podStartSLOduration=1.7597720749999999 podStartE2EDuration="1.759772075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:52.757090334 +0000 UTC m=+1954.959968395" watchObservedRunningTime="2026-11-02 15:30:52.759772075 +0000 UTC m=+1954.962650135" Nov 02 15:31:00.289133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:00.289088 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg"] Nov 02 15:31:00.289551 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:00.289304 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" podUID="1fb515d1-ae20-4c42-bba2-a719b3049927" containerName="istio-proxy" containerID="cri-o://2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f" gracePeriod=30 Nov 02 15:31:01.841985 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:01.841945 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:31:03.705366 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:03.705335 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:31:05.564966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.564905 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:31:05.656343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656317 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-kube-api-access-grsrm\" (UniqueName: \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-kube-api-access-grsrm\") pod \"1fb515d1-ae20-4c42-bba2-a719b3049927\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " Nov 02 15:31:05.656507 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656365 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-socket\") pod \"1fb515d1-ae20-4c42-bba2-a719b3049927\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " Nov 02 15:31:05.656507 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656394 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-certs\") pod \"1fb515d1-ae20-4c42-bba2-a719b3049927\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " Nov 02 15:31:05.656507 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656439 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-credential-socket\") pod \"1fb515d1-ae20-4c42-bba2-a719b3049927\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " Nov 02 15:31:05.656507 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656468 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-podinfo\") pod \"1fb515d1-ae20-4c42-bba2-a719b3049927\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " Nov 02 15:31:05.656740 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656575 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-data\") pod \"1fb515d1-ae20-4c42-bba2-a719b3049927\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " Nov 02 15:31:05.656740 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656619 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-token\" (UniqueName: \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-token\") pod \"1fb515d1-ae20-4c42-bba2-a719b3049927\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " Nov 02 15:31:05.656740 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656678 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-envoy\") pod \"1fb515d1-ae20-4c42-bba2-a719b3049927\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " Nov 02 15:31:05.656892 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656731 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-credential-socket" pod "1fb515d1-ae20-4c42-bba2-a719b3049927" (UID: "1fb515d1-ae20-4c42-bba2-a719b3049927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:05.656892 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656738 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-socket" pod "1fb515d1-ae20-4c42-bba2-a719b3049927" (UID: "1fb515d1-ae20-4c42-bba2-a719b3049927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:05.656892 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656759 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb515d1-ae20-4c42-bba2-a719b3049927-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb515d1-ae20-4c42-bba2-a719b3049927-istiod-ca-cert\") pod \"1fb515d1-ae20-4c42-bba2-a719b3049927\" (UID: \"1fb515d1-ae20-4c42-bba2-a719b3049927\") " Nov 02 15:31:05.657532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656746 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-certs" pod "1fb515d1-ae20-4c42-bba2-a719b3049927" (UID: "1fb515d1-ae20-4c42-bba2-a719b3049927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:05.657532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.656966 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-data" pod "1fb515d1-ae20-4c42-bba2-a719b3049927" (UID: "1fb515d1-ae20-4c42-bba2-a719b3049927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:05.657532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.657435 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb515d1-ae20-4c42-bba2-a719b3049927-istiod-ca-cert" pod "1fb515d1-ae20-4c42-bba2-a719b3049927" (UID: "1fb515d1-ae20-4c42-bba2-a719b3049927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:05.657754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.657670 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:05.657754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.657699 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:05.657754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.657716 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:05.657754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.657729 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:05.659107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.659079 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-kube-api-access-grsrm" pod "1fb515d1-ae20-4c42-bba2-a719b3049927" (UID: "1fb515d1-ae20-4c42-bba2-a719b3049927"). InnerVolumeSpecName "kube-api-access-grsrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:05.659783 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.659753 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-token" pod "1fb515d1-ae20-4c42-bba2-a719b3049927" (UID: "1fb515d1-ae20-4c42-bba2-a719b3049927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:05.659870 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.659854 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-envoy" pod "1fb515d1-ae20-4c42-bba2-a719b3049927" (UID: "1fb515d1-ae20-4c42-bba2-a719b3049927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:05.659920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.659896 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-podinfo" pod "1fb515d1-ae20-4c42-bba2-a719b3049927" (UID: "1fb515d1-ae20-4c42-bba2-a719b3049927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:31:05.758739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.758706 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:05.758739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.758733 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:05.758739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.758742 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb515d1-ae20-4c42-bba2-a719b3049927-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:05.758963 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.758753 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grsrm\" (UniqueName: \"kubernetes.io/projected/1fb515d1-ae20-4c42-bba2-a719b3049927-kube-api-access-grsrm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:05.758963 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.758764 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb515d1-ae20-4c42-bba2-a719b3049927-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:05.798930 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.798899 5476 generic.go:356] "Generic (PLEG): container finished" podID="1fb515d1-ae20-4c42-bba2-a719b3049927" containerID="2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f" exitCode=0 Nov 02 15:31:05.799122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.798963 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" Nov 02 15:31:05.799122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.799005 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" event={"ID":"1fb515d1-ae20-4c42-bba2-a719b3049927","Type":"ContainerDied","Data":"2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f"} Nov 02 15:31:05.799122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.799065 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg" event={"ID":"1fb515d1-ae20-4c42-bba2-a719b3049927","Type":"ContainerDied","Data":"dac87d1ad0cbfee0ed64366c9c9178530319ce92c76f5140719e419c0c2fb247"} Nov 02 15:31:05.799122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.799099 5476 scope.go:122] "RemoveContainer" containerID="2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f" Nov 02 15:31:05.813889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.813863 5476 scope.go:122] "RemoveContainer" containerID="2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f" Nov 02 15:31:05.814245 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:31:05.814222 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f\": container with ID starting with 2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f not found: ID does not exist" containerID="2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f" Nov 02 15:31:05.814341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.814257 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f"} err="failed to get container status \"2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f\": rpc error: code = NotFound desc = could not find container \"2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f\": container with ID starting with 2e4079bc25088de13b3afbcc58672fb40c84473feb071fa868e476a4162b625f not found: ID does not exist" Nov 02 15:31:05.955563 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.955529 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg"] Nov 02 15:31:05.960464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:05.960438 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-8z4fg"] Nov 02 15:31:06.236426 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.236345 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x"] Nov 02 15:31:06.237193 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.237175 5476 cpu_manager.go:414] "containerMap: removing container" podUID="1fb515d1-ae20-4c42-bba2-a719b3049927" containerName="istio-proxy" Nov 02 15:31:06.237193 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.237195 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb515d1-ae20-4c42-bba2-a719b3049927" containerName="istio-proxy" Nov 02 15:31:06.237322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.237314 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb515d1-ae20-4c42-bba2-a719b3049927" containerName="istio-proxy" Nov 02 15:31:06.242121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.242101 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.265128 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.265092 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-openshift-service-ca\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.265259 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.265138 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-maas-api-tls\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.265308 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.265262 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm6ft\" (UniqueName: \"kubernetes.io/projected/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-kube-api-access-pm6ft\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.265344 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.265331 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-metrics-certs\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.293202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.293171 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0de358-serving-cert\"" Nov 02 15:31:06.299044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.299020 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x"] Nov 02 15:31:06.366592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.366569 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-maas-api-tls\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.366713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.366616 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm6ft\" (UniqueName: \"kubernetes.io/projected/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-kube-api-access-pm6ft\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.366713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.366670 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-metrics-certs\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.366788 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:31:06.366719 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-serving-cert: secret "maas-api-e2e-api-a-0de358-serving-cert" not found Nov 02 15:31:06.366822 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:31:06.366799 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-maas-api-tls podName:90d7d3e9-465f-4da2-9ad8-4702fe5f9af5 nodeName:}" failed. No retries permitted until 2026-11-02 15:31:06.866782749 +0000 UTC m=+1969.069660787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-maas-api-tls") pod "maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" (UID: "90d7d3e9-465f-4da2-9ad8-4702fe5f9af5") : secret "maas-api-e2e-api-a-0de358-serving-cert" not found Nov 02 15:31:06.366912 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.366896 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-openshift-service-ca\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.367538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.367514 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-openshift-service-ca\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.369246 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.369227 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-metrics-certs\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.427053 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.427028 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm6ft\" (UniqueName: \"kubernetes.io/projected/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-kube-api-access-pm6ft\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.873400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.873367 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-maas-api-tls\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:06.875924 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:06.875903 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-maas-api-tls\") pod \"maas-api-e2e-api-a-0de358-cbb55bfb8-x262x\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:07.155520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:07.155442 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:07.535100 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:07.535013 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x"] Nov 02 15:31:07.817429 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:07.817347 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" event={"ID":"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5","Type":"ContainerStarted","Data":"a84ba92b8f959cb71144f80eabc02f70ac586036ca733f5bcd4d5e36d4344e16"} Nov 02 15:31:07.924083 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:07.924047 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb515d1-ae20-4c42-bba2-a719b3049927" path="/var/lib/kubelet/pods/1fb515d1-ae20-4c42-bba2-a719b3049927/volumes" Nov 02 15:31:08.826887 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:08.826795 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" event={"ID":"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5","Type":"ContainerStarted","Data":"e95ffcec1c67bff026fc69531fa0440dd8af519f974176335497489d2507ea22"} Nov 02 15:31:08.826887 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:08.826883 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:08.938930 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:08.938874 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" podStartSLOduration=1.919268208 podStartE2EDuration="2.938859669s" totalImagesPullingTime="1.019591461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:08.937865249 +0000 UTC m=+1971.140743310" watchObservedRunningTime="2026-11-02 15:31:08.938859669 +0000 UTC m=+1971.141737731" Nov 02 15:31:14.840064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:14.840033 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:31:20.146729 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:31:20.146693 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:31:38.978721 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:38.978634 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97"] Nov 02 15:31:38.983081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:38.983057 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:31:39.039400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:39.039368 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42d5004-cd64-4077-a706-8e7920be3392-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97\" (UID: \"a42d5004-cd64-4077-a706-8e7920be3392\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:31:39.045425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:39.045395 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwfc9\" (UniqueName: \"kubernetes.io/projected/a42d5004-cd64-4077-a706-8e7920be3392-kube-api-access-cwfc9\") pod \"payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97\" (UID: \"a42d5004-cd64-4077-a706-8e7920be3392\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:31:39.088757 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:39.088369 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-adf096\"" Nov 02 15:31:39.163420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:39.163368 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwfc9\" (UniqueName: \"kubernetes.io/projected/a42d5004-cd64-4077-a706-8e7920be3392-kube-api-access-cwfc9\") pod \"payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97\" (UID: \"a42d5004-cd64-4077-a706-8e7920be3392\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:31:39.164610 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:39.164558 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42d5004-cd64-4077-a706-8e7920be3392-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97\" (UID: \"a42d5004-cd64-4077-a706-8e7920be3392\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:31:39.165922 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:39.165883 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42d5004-cd64-4077-a706-8e7920be3392-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97\" (UID: \"a42d5004-cd64-4077-a706-8e7920be3392\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:31:39.171207 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:39.166914 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-adf096-dockercfg-dcp8l\"" Nov 02 15:31:39.172206 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:39.172174 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97"] Nov 02 15:31:39.229256 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:39.229177 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwfc9\" (UniqueName: \"kubernetes.io/projected/a42d5004-cd64-4077-a706-8e7920be3392-kube-api-access-cwfc9\") pod \"payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97\" (UID: \"a42d5004-cd64-4077-a706-8e7920be3392\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:31:39.315167 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:39.315129 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:31:39.705367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:39.705324 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97"] Nov 02 15:31:39.711470 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:31:39.711441 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42d5004_cd64_4077_a706_8e7920be3392.slice/crio-53216c4b781a7c7d45a87dd37e48df46f89e5a17a6245ca493da986bc5c532d2 WatchSource:0}: Error finding container 53216c4b781a7c7d45a87dd37e48df46f89e5a17a6245ca493da986bc5c532d2: Status 404 returned error can't find the container with id 53216c4b781a7c7d45a87dd37e48df46f89e5a17a6245ca493da986bc5c532d2 Nov 02 15:31:40.204080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:40.204027 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" event={"ID":"a42d5004-cd64-4077-a706-8e7920be3392","Type":"ContainerStarted","Data":"76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7"} Nov 02 15:31:40.204528 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:40.204093 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" event={"ID":"a42d5004-cd64-4077-a706-8e7920be3392","Type":"ContainerStarted","Data":"53216c4b781a7c7d45a87dd37e48df46f89e5a17a6245ca493da986bc5c532d2"} Nov 02 15:31:40.204528 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:40.204256 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:31:45.706420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:45.706354 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" podStartSLOduration=7.706334777 podStartE2EDuration="7.706334777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:40.317149551 +0000 UTC m=+2002.520027612" watchObservedRunningTime="2026-11-02 15:31:45.706334777 +0000 UTC m=+2007.909212838" Nov 02 15:31:45.711137 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:45.711106 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq"] Nov 02 15:31:45.711382 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:45.711357 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" containerID="cri-o://2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e" gracePeriod=30 Nov 02 15:31:47.295549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:47.295500 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2"] Nov 02 15:31:47.295925 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:47.295796 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" podUID="93564a79-a336-4637-8182-ce90ca9b9938" containerName="istio-proxy" containerID="cri-o://294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5" gracePeriod=30 Nov 02 15:31:49.191778 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:49.191745 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb"] Nov 02 15:31:49.192299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:49.192025 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" containerID="cri-o://decdd28a346334c8dcd0f1f19ff3b2aee165176bec9f9a575bdffae832b13295" gracePeriod=30 Nov 02 15:31:49.277135 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:49.277098 5476 generic.go:356] "Generic (PLEG): container finished" podID="9af3940e-fa13-4569-9951-a338c215ba0d" containerID="5d57848a937bff863cfed4a1e34767560641123b211c2c3c612c21211555d498" exitCode=0 Nov 02 15:31:49.277331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:49.277147 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" event={"ID":"9af3940e-fa13-4569-9951-a338c215ba0d","Type":"ContainerDied","Data":"5d57848a937bff863cfed4a1e34767560641123b211c2c3c612c21211555d498"} Nov 02 15:31:49.277738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:49.277719 5476 scope.go:122] "RemoveContainer" containerID="5d57848a937bff863cfed4a1e34767560641123b211c2c3c612c21211555d498" Nov 02 15:31:49.279899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:49.279861 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:49.280023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:49.279947 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:49.280023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:49.280012 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:50.147534 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:31:50.147494 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:31:50.308230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:50.308183 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" event={"ID":"9af3940e-fa13-4569-9951-a338c215ba0d","Type":"ContainerStarted","Data":"dc1c072823c9a0254e2091353f7483d2e35c5f96809c1ffaeb46fc4bfc79c071"} Nov 02 15:31:50.664113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:50.664086 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:31:50.664415 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:50.664389 5476 patch_prober.go:29] interesting pod/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 15:31:50.664531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:50.664461 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 15:31:51.038979 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.038952 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:31:51.211584 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.211527 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-envoy\") pod \"d62a3481-0bbf-4d16-83f0-524d84b5964c\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " Nov 02 15:31:51.211779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.211596 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-credential-socket\") pod \"d62a3481-0bbf-4d16-83f0-524d84b5964c\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " Nov 02 15:31:51.211779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.211647 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-kube-api-access-qsh4k\" (UniqueName: \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-kube-api-access-qsh4k\") pod \"d62a3481-0bbf-4d16-83f0-524d84b5964c\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " Nov 02 15:31:51.211779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.211750 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-podinfo\") pod \"d62a3481-0bbf-4d16-83f0-524d84b5964c\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " Nov 02 15:31:51.211973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.211784 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62a3481-0bbf-4d16-83f0-524d84b5964c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62a3481-0bbf-4d16-83f0-524d84b5964c-istiod-ca-cert\") pod \"d62a3481-0bbf-4d16-83f0-524d84b5964c\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " Nov 02 15:31:51.211973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.211921 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-certs\") pod \"d62a3481-0bbf-4d16-83f0-524d84b5964c\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " Nov 02 15:31:51.212104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.211986 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-credential-socket" pod "d62a3481-0bbf-4d16-83f0-524d84b5964c" (UID: "d62a3481-0bbf-4d16-83f0-524d84b5964c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:51.212221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.212198 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-certs" pod "d62a3481-0bbf-4d16-83f0-524d84b5964c" (UID: "d62a3481-0bbf-4d16-83f0-524d84b5964c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:51.212532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.212509 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-socket\") pod \"d62a3481-0bbf-4d16-83f0-524d84b5964c\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " Nov 02 15:31:51.212532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.212519 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62a3481-0bbf-4d16-83f0-524d84b5964c-istiod-ca-cert" pod "d62a3481-0bbf-4d16-83f0-524d84b5964c" (UID: "d62a3481-0bbf-4d16-83f0-524d84b5964c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:51.212681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.212579 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-data\") pod \"d62a3481-0bbf-4d16-83f0-524d84b5964c\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " Nov 02 15:31:51.212764 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.212744 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-token\" (UniqueName: \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-token\") pod \"d62a3481-0bbf-4d16-83f0-524d84b5964c\" (UID: \"d62a3481-0bbf-4d16-83f0-524d84b5964c\") " Nov 02 15:31:51.213535 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.213506 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.213647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.213539 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.213647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.213554 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d62a3481-0bbf-4d16-83f0-524d84b5964c-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.217074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.214654 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-data" pod "d62a3481-0bbf-4d16-83f0-524d84b5964c" (UID: "d62a3481-0bbf-4d16-83f0-524d84b5964c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:51.217074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.215202 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-socket" pod "d62a3481-0bbf-4d16-83f0-524d84b5964c" (UID: "d62a3481-0bbf-4d16-83f0-524d84b5964c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:51.217074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.215566 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:31:51.218183 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.218116 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-envoy" pod "d62a3481-0bbf-4d16-83f0-524d84b5964c" (UID: "d62a3481-0bbf-4d16-83f0-524d84b5964c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:51.218183 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.218157 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-podinfo" pod "d62a3481-0bbf-4d16-83f0-524d84b5964c" (UID: "d62a3481-0bbf-4d16-83f0-524d84b5964c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:31:51.218334 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.218210 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-kube-api-access-qsh4k" pod "d62a3481-0bbf-4d16-83f0-524d84b5964c" (UID: "d62a3481-0bbf-4d16-83f0-524d84b5964c"). InnerVolumeSpecName "kube-api-access-qsh4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:51.218477 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.218445 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-token" pod "d62a3481-0bbf-4d16-83f0-524d84b5964c" (UID: "d62a3481-0bbf-4d16-83f0-524d84b5964c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:51.314597 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.314523 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.314597 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.314549 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsh4k\" (UniqueName: \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-kube-api-access-qsh4k\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.314597 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.314566 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.314597 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.314577 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.314597 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.314586 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.314597 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.314595 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d62a3481-0bbf-4d16-83f0-524d84b5964c-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.317110 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.317080 5476 generic.go:356] "Generic (PLEG): container finished" podID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerID="2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e" exitCode=0 Nov 02 15:31:51.317264 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.317118 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" event={"ID":"d62a3481-0bbf-4d16-83f0-524d84b5964c","Type":"ContainerDied","Data":"2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e"} Nov 02 15:31:51.317264 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.317146 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" Nov 02 15:31:51.317264 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.317171 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq" event={"ID":"d62a3481-0bbf-4d16-83f0-524d84b5964c","Type":"ContainerDied","Data":"db26a69f6cc5493bf6c5252a2eda24038b0c3993f961f327a6acb9fd9862155e"} Nov 02 15:31:51.317264 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.317195 5476 scope.go:122] "RemoveContainer" containerID="2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e" Nov 02 15:31:51.329280 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.329223 5476 scope.go:122] "RemoveContainer" containerID="2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e" Nov 02 15:31:51.329544 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:31:51.329516 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e\": container with ID starting with 2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e not found: ID does not exist" containerID="2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e" Nov 02 15:31:51.329667 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.329544 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e"} err="failed to get container status \"2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e\": rpc error: code = NotFound desc = could not find container \"2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e\": container with ID starting with 2904e6b494b0a191f32d0227e05e259bdcdef8e47e6b6d1955c8c5544351f22e not found: ID does not exist" Nov 02 15:31:51.498820 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.498787 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq"] Nov 02 15:31:51.503966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.503940 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-78nkq"] Nov 02 15:31:51.664570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.664532 5476 patch_prober.go:29] interesting pod/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 15:31:51.664738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.664589 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 15:31:51.924934 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:51.924858 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" path="/var/lib/kubelet/pods/d62a3481-0bbf-4d16-83f0-524d84b5964c/volumes" Nov 02 15:31:52.664465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:52.664431 5476 patch_prober.go:29] interesting pod/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 15:31:52.664971 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:52.664491 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 15:31:52.810240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:52.810211 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Nov 02 15:31:52.810415 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:52.810261 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" podUID="93564a79-a336-4637-8182-ce90ca9b9938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Nov 02 15:31:53.034615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.034584 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:31:53.140950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.140916 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93564a79-a336-4637-8182-ce90ca9b9938-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93564a79-a336-4637-8182-ce90ca9b9938-istiod-ca-cert\") pod \"93564a79-a336-4637-8182-ce90ca9b9938\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " Nov 02 15:31:53.141154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.141046 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-kube-api-access-q2bx8\" (UniqueName: \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-kube-api-access-q2bx8\") pod \"93564a79-a336-4637-8182-ce90ca9b9938\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " Nov 02 15:31:53.141230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.141149 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93564a79-a336-4637-8182-ce90ca9b9938-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93564a79-a336-4637-8182-ce90ca9b9938-istio-podinfo\") pod \"93564a79-a336-4637-8182-ce90ca9b9938\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " Nov 02 15:31:53.141289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.141233 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-istio-token\" (UniqueName: \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-istio-token\") pod \"93564a79-a336-4637-8182-ce90ca9b9938\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " Nov 02 15:31:53.141352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.141312 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-credential-socket\") pod \"93564a79-a336-4637-8182-ce90ca9b9938\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " Nov 02 15:31:53.141352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.141337 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93564a79-a336-4637-8182-ce90ca9b9938-istiod-ca-cert" pod "93564a79-a336-4637-8182-ce90ca9b9938" (UID: "93564a79-a336-4637-8182-ce90ca9b9938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:53.141470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.141353 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-socket\") pod \"93564a79-a336-4637-8182-ce90ca9b9938\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " Nov 02 15:31:53.141470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.141422 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-envoy\") pod \"93564a79-a336-4637-8182-ce90ca9b9938\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " Nov 02 15:31:53.141470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.141457 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-certs\") pod \"93564a79-a336-4637-8182-ce90ca9b9938\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " Nov 02 15:31:53.142202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.141488 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-data\") pod \"93564a79-a336-4637-8182-ce90ca9b9938\" (UID: \"93564a79-a336-4637-8182-ce90ca9b9938\") " Nov 02 15:31:53.142202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.141865 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-credential-socket" pod "93564a79-a336-4637-8182-ce90ca9b9938" (UID: "93564a79-a336-4637-8182-ce90ca9b9938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:53.142202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.142183 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-certs" pod "93564a79-a336-4637-8182-ce90ca9b9938" (UID: "93564a79-a336-4637-8182-ce90ca9b9938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:53.142483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.142350 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-data" pod "93564a79-a336-4637-8182-ce90ca9b9938" (UID: "93564a79-a336-4637-8182-ce90ca9b9938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:53.142483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.142371 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:53.142483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.142388 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:53.142483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.142400 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93564a79-a336-4637-8182-ce90ca9b9938-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:53.142690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.142487 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-socket" pod "93564a79-a336-4637-8182-ce90ca9b9938" (UID: "93564a79-a336-4637-8182-ce90ca9b9938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:53.144056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.144025 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-istio-token" pod "93564a79-a336-4637-8182-ce90ca9b9938" (UID: "93564a79-a336-4637-8182-ce90ca9b9938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:53.144221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.144186 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-kube-api-access-q2bx8" pod "93564a79-a336-4637-8182-ce90ca9b9938" (UID: "93564a79-a336-4637-8182-ce90ca9b9938"). InnerVolumeSpecName "kube-api-access-q2bx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:53.145090 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.145066 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93564a79-a336-4637-8182-ce90ca9b9938-istio-podinfo" pod "93564a79-a336-4637-8182-ce90ca9b9938" (UID: "93564a79-a336-4637-8182-ce90ca9b9938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:31:53.145577 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.145549 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-envoy" pod "93564a79-a336-4637-8182-ce90ca9b9938" (UID: "93564a79-a336-4637-8182-ce90ca9b9938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:53.243280 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.243244 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2bx8\" (UniqueName: \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-kube-api-access-q2bx8\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:53.243280 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.243279 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93564a79-a336-4637-8182-ce90ca9b9938-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:53.243280 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.243290 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93564a79-a336-4637-8182-ce90ca9b9938-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:53.243521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.243300 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:53.243521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.243309 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:53.243521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.243319 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93564a79-a336-4637-8182-ce90ca9b9938-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:53.338153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.338048 5476 generic.go:356] "Generic (PLEG): container finished" podID="93564a79-a336-4637-8182-ce90ca9b9938" containerID="294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5" exitCode=0 Nov 02 15:31:53.338153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.338114 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" Nov 02 15:31:53.338153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.338140 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" event={"ID":"93564a79-a336-4637-8182-ce90ca9b9938","Type":"ContainerDied","Data":"294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5"} Nov 02 15:31:53.338432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.338203 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2" event={"ID":"93564a79-a336-4637-8182-ce90ca9b9938","Type":"ContainerDied","Data":"df9552317a27f5ddc74981aad4d1a3a2daa67fabf4a3fb25a9983ff427a18e94"} Nov 02 15:31:53.338432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.338232 5476 scope.go:122] "RemoveContainer" containerID="294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5" Nov 02 15:31:53.352874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.352848 5476 scope.go:122] "RemoveContainer" containerID="294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5" Nov 02 15:31:53.353152 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:31:53.353131 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5\": container with ID starting with 294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5 not found: ID does not exist" containerID="294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5" Nov 02 15:31:53.353224 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.353159 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5"} err="failed to get container status \"294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5\": rpc error: code = NotFound desc = could not find container \"294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5\": container with ID starting with 294d5356969db581a461f9f6996e47d16f723b91ff2d3610d5f6121aef4656f5 not found: ID does not exist" Nov 02 15:31:53.503041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.502707 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2"] Nov 02 15:31:53.506527 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.506497 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-jrzq2"] Nov 02 15:31:53.664555 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.664520 5476 patch_prober.go:29] interesting pod/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 15:31:53.665071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.664570 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 15:31:53.929059 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:53.928309 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93564a79-a336-4637-8182-ce90ca9b9938" path="/var/lib/kubelet/pods/93564a79-a336-4637-8182-ce90ca9b9938/volumes" Nov 02 15:31:54.349137 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.349098 5476 generic.go:356] "Generic (PLEG): container finished" podID="779804e1-968d-464a-83f3-648de3d24208" containerID="decdd28a346334c8dcd0f1f19ff3b2aee165176bec9f9a575bdffae832b13295" exitCode=0 Nov 02 15:31:54.349317 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.349299 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" event={"ID":"779804e1-968d-464a-83f3-648de3d24208","Type":"ContainerDied","Data":"decdd28a346334c8dcd0f1f19ff3b2aee165176bec9f9a575bdffae832b13295"} Nov 02 15:31:54.534806 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.534780 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:31:54.562718 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.562685 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-istio-token\" (UniqueName: \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-istio-token\") pod \"779804e1-968d-464a-83f3-648de3d24208\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " Nov 02 15:31:54.562874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.562750 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-credential-socket\") pod \"779804e1-968d-464a-83f3-648de3d24208\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " Nov 02 15:31:54.562874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.562831 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-data\") pod \"779804e1-968d-464a-83f3-648de3d24208\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " Nov 02 15:31:54.562975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.562959 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-socket\") pod \"779804e1-968d-464a-83f3-648de3d24208\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " Nov 02 15:31:54.563074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.563058 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-certs\") pod \"779804e1-968d-464a-83f3-648de3d24208\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " Nov 02 15:31:54.563131 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.563121 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/779804e1-968d-464a-83f3-648de3d24208-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779804e1-968d-464a-83f3-648de3d24208-istiod-ca-cert\") pod \"779804e1-968d-464a-83f3-648de3d24208\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " Nov 02 15:31:54.563194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.563164 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-kube-api-access-2p5nw\" (UniqueName: \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-kube-api-access-2p5nw\") pod \"779804e1-968d-464a-83f3-648de3d24208\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " Nov 02 15:31:54.563194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.563161 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-data" pod "779804e1-968d-464a-83f3-648de3d24208" (UID: "779804e1-968d-464a-83f3-648de3d24208"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:54.563308 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.563282 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-envoy\") pod \"779804e1-968d-464a-83f3-648de3d24208\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " Nov 02 15:31:54.563366 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.563316 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/779804e1-968d-464a-83f3-648de3d24208-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779804e1-968d-464a-83f3-648de3d24208-istio-podinfo\") pod \"779804e1-968d-464a-83f3-648de3d24208\" (UID: \"779804e1-968d-464a-83f3-648de3d24208\") " Nov 02 15:31:54.563427 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.563357 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-socket" pod "779804e1-968d-464a-83f3-648de3d24208" (UID: "779804e1-968d-464a-83f3-648de3d24208"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:54.563427 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.563385 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-certs" pod "779804e1-968d-464a-83f3-648de3d24208" (UID: "779804e1-968d-464a-83f3-648de3d24208"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:54.563768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.563743 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/779804e1-968d-464a-83f3-648de3d24208-istiod-ca-cert" pod "779804e1-968d-464a-83f3-648de3d24208" (UID: "779804e1-968d-464a-83f3-648de3d24208"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:54.563874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.563840 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-credential-socket" pod "779804e1-968d-464a-83f3-648de3d24208" (UID: "779804e1-968d-464a-83f3-648de3d24208"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:54.564094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.564074 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.564183 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.564098 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.564183 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.564115 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.564183 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.564128 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/779804e1-968d-464a-83f3-648de3d24208-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.564183 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.564141 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.566363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.566243 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-istio-token" pod "779804e1-968d-464a-83f3-648de3d24208" (UID: "779804e1-968d-464a-83f3-648de3d24208"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:54.566363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.566324 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/779804e1-968d-464a-83f3-648de3d24208-istio-podinfo" pod "779804e1-968d-464a-83f3-648de3d24208" (UID: "779804e1-968d-464a-83f3-648de3d24208"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:31:54.566363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.566329 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-kube-api-access-2p5nw" pod "779804e1-968d-464a-83f3-648de3d24208" (UID: "779804e1-968d-464a-83f3-648de3d24208"). InnerVolumeSpecName "kube-api-access-2p5nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:54.566568 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.566545 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-envoy" pod "779804e1-968d-464a-83f3-648de3d24208" (UID: "779804e1-968d-464a-83f3-648de3d24208"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:54.663844 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.663795 5476 patch_prober.go:29] interesting pod/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Nov 02 15:31:54.664058 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.663866 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Nov 02 15:31:54.666113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.666070 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/779804e1-968d-464a-83f3-648de3d24208-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.666113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.666101 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/779804e1-968d-464a-83f3-648de3d24208-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.666113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.666114 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.666524 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:54.666127 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p5nw\" (UniqueName: \"kubernetes.io/projected/779804e1-968d-464a-83f3-648de3d24208-kube-api-access-2p5nw\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:55.357449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:55.357420 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" Nov 02 15:31:55.357678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:55.357450 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb" event={"ID":"779804e1-968d-464a-83f3-648de3d24208","Type":"ContainerDied","Data":"838b29c8d1e6843dd4048dceb9c9235c0e75404c08cdb58f8cfad133aa327382"} Nov 02 15:31:55.357678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:55.357518 5476 scope.go:122] "RemoveContainer" containerID="decdd28a346334c8dcd0f1f19ff3b2aee165176bec9f9a575bdffae832b13295" Nov 02 15:31:55.515597 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:55.515559 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb"] Nov 02 15:31:55.521675 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:55.521648 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-crpwb"] Nov 02 15:31:55.669202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:55.669120 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:31:55.669648 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:55.669559 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:31:55.670395 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:55.670371 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:31:55.925680 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:55.925603 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="779804e1-968d-464a-83f3-648de3d24208" path="/var/lib/kubelet/pods/779804e1-968d-464a-83f3-648de3d24208/volumes" Nov 02 15:31:57.222669 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.222635 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x"] Nov 02 15:31:57.223561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.223540 5476 cpu_manager.go:414] "containerMap: removing container" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" Nov 02 15:31:57.223611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.223565 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" Nov 02 15:31:57.223653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.223614 5476 cpu_manager.go:414] "containerMap: removing container" podUID="93564a79-a336-4637-8182-ce90ca9b9938" containerName="istio-proxy" Nov 02 15:31:57.223653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.223622 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93564a79-a336-4637-8182-ce90ca9b9938" containerName="istio-proxy" Nov 02 15:31:57.223736 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.223650 5476 cpu_manager.go:414] "containerMap: removing container" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" Nov 02 15:31:57.223736 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.223659 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" Nov 02 15:31:57.223834 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.223802 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62a3481-0bbf-4d16-83f0-524d84b5964c" containerName="istio-proxy" Nov 02 15:31:57.223834 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.223823 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="93564a79-a336-4637-8182-ce90ca9b9938" containerName="istio-proxy" Nov 02 15:31:57.223927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.223836 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="779804e1-968d-464a-83f3-648de3d24208" containerName="istio-proxy" Nov 02 15:31:57.228770 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.228750 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.277824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.277789 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-5afebc-openshift-default-dockercfg-57h7j\"" Nov 02 15:31:57.314531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.314500 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.314686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.314534 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2042e87-2978-48d1-a54f-a6c785cc68dd-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.314686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.314648 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.314814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.314793 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.314913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.314891 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.314977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.314939 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.315051 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.315033 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jcf\" (UniqueName: \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-kube-api-access-j4jcf\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.315117 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.315076 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.315186 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.315167 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.330196 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.330170 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x"] Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.416733 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.416790 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.416817 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.416854 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jcf\" (UniqueName: \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-kube-api-access-j4jcf\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.416893 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.416964 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.417121 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.417145 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2042e87-2978-48d1-a54f-a6c785cc68dd-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.417316 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.417800 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.417920 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.418250 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.418320 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2042e87-2978-48d1-a54f-a6c785cc68dd-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.418465 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.419859 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.424084 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.423929 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.474727 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.474657 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.474873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.474745 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jcf\" (UniqueName: \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-kube-api-access-j4jcf\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.542843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.542575 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:57.980962 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.980725 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x"] Nov 02 15:31:57.985334 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.985144 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:57.985429 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.985368 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:57.985429 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:57.985396 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:58.403797 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:58.403758 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" event={"ID":"a2042e87-2978-48d1-a54f-a6c785cc68dd","Type":"ContainerStarted","Data":"4bc155a3dd402aab33edb9b7de65971f3b12072da0087f6a331c1a690c434a65"} Nov 02 15:31:58.404371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:58.403822 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" event={"ID":"a2042e87-2978-48d1-a54f-a6c785cc68dd","Type":"ContainerStarted","Data":"ee3e18c2bd344f42eacca2649552cd80b57a078065a88e1f4e3b1dbc1a23eafa"} Nov 02 15:31:58.518373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:58.518322 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" podStartSLOduration=1.518308206 podStartE2EDuration="1.518308206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:58.51745725 +0000 UTC m=+2020.720335310" watchObservedRunningTime="2026-11-02 15:31:58.518308206 +0000 UTC m=+2020.721186275" Nov 02 15:31:58.543292 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:58.543266 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:31:59.549122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:31:59.549095 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:32:00.419736 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:00.419706 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:32:00.420687 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:00.420662 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:32:01.965806 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:01.965767 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:32:04.432038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:04.428093 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9"] Nov 02 15:32:04.432038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:04.428394 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerName="istio-proxy" containerID="cri-o://9c3b0f25541ffecea7e8a236f2deb2ffb6642dd141c86d3012d25a0ea9a7c09f" gracePeriod=30 Nov 02 15:32:05.347017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.324962 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z"] Nov 02 15:32:05.352956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.352927 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:32:05.391328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.391293 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z"] Nov 02 15:32:05.432052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.430944 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-5afebc\"" Nov 02 15:32:05.432052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.431102 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-5afebc-dockercfg-p2sg9\"" Nov 02 15:32:05.492642 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.492602 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj"] Nov 02 15:32:05.646043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.569401 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b869364-017a-4cdd-8501-c68fcee1d73b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z\" (UID: \"9b869364-017a-4cdd-8501-c68fcee1d73b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:32:05.648671 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.648643 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx8lb\" (UniqueName: \"kubernetes.io/projected/9b869364-017a-4cdd-8501-c68fcee1d73b-kube-api-access-tx8lb\") pod \"payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z\" (UID: \"9b869364-017a-4cdd-8501-c68fcee1d73b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:32:05.656551 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.656532 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:32:05.676630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.676601 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj"] Nov 02 15:32:05.676878 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.676851 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:32:05.695053 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.687079 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kk2m9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:32:05.695053 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.687137 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podUID="afd13c1e-9b91-4be5-92ed-ceea14f0203c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:32:05.751182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.751133 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6g4\" (UniqueName: \"kubernetes.io/projected/b785048a-948c-4334-a2cb-c12ff8990d24-kube-api-access-7c6g4\") pod \"payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj\" (UID: \"b785048a-948c-4334-a2cb-c12ff8990d24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:32:05.751681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.751608 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b785048a-948c-4334-a2cb-c12ff8990d24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj\" (UID: \"b785048a-948c-4334-a2cb-c12ff8990d24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:32:05.752068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.752044 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b869364-017a-4cdd-8501-c68fcee1d73b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z\" (UID: \"9b869364-017a-4cdd-8501-c68fcee1d73b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:32:05.753212 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.753192 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx8lb\" (UniqueName: \"kubernetes.io/projected/9b869364-017a-4cdd-8501-c68fcee1d73b-kube-api-access-tx8lb\") pod \"payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z\" (UID: \"9b869364-017a-4cdd-8501-c68fcee1d73b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:32:05.754941 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.754913 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b869364-017a-4cdd-8501-c68fcee1d73b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z\" (UID: \"9b869364-017a-4cdd-8501-c68fcee1d73b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:32:05.815671 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.812773 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx8lb\" (UniqueName: \"kubernetes.io/projected/9b869364-017a-4cdd-8501-c68fcee1d73b-kube-api-access-tx8lb\") pod \"payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z\" (UID: \"9b869364-017a-4cdd-8501-c68fcee1d73b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:32:05.855691 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.855660 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6g4\" (UniqueName: \"kubernetes.io/projected/b785048a-948c-4334-a2cb-c12ff8990d24-kube-api-access-7c6g4\") pod \"payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj\" (UID: \"b785048a-948c-4334-a2cb-c12ff8990d24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:32:05.855868 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.855726 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b785048a-948c-4334-a2cb-c12ff8990d24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj\" (UID: \"b785048a-948c-4334-a2cb-c12ff8990d24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:32:05.856761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.856734 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b785048a-948c-4334-a2cb-c12ff8990d24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj\" (UID: \"b785048a-948c-4334-a2cb-c12ff8990d24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:32:05.914067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:05.913963 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6g4\" (UniqueName: \"kubernetes.io/projected/b785048a-948c-4334-a2cb-c12ff8990d24-kube-api-access-7c6g4\") pod \"payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj\" (UID: \"b785048a-948c-4334-a2cb-c12ff8990d24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:32:06.009184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.009153 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:32:06.023422 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.023398 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:32:06.408268 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.408239 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z"] Nov 02 15:32:06.412318 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:32:06.412291 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b869364_017a_4cdd_8501_c68fcee1d73b.slice/crio-bb41fddf49dd6d37dc4744c6dc836ee1ec337aa34dae29ef41e6994fafb6e108 WatchSource:0}: Error finding container bb41fddf49dd6d37dc4744c6dc836ee1ec337aa34dae29ef41e6994fafb6e108: Status 404 returned error can't find the container with id bb41fddf49dd6d37dc4744c6dc836ee1ec337aa34dae29ef41e6994fafb6e108 Nov 02 15:32:06.455237 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.455210 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj"] Nov 02 15:32:06.471261 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:32:06.471226 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb785048a_948c_4334_a2cb_c12ff8990d24.slice/crio-b9d262174dd27d380e6bfbbeeebe4e331762bce6eae4f2288d2014e05743a4c7 WatchSource:0}: Error finding container b9d262174dd27d380e6bfbbeeebe4e331762bce6eae4f2288d2014e05743a4c7: Status 404 returned error can't find the container with id b9d262174dd27d380e6bfbbeeebe4e331762bce6eae4f2288d2014e05743a4c7 Nov 02 15:32:06.647023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.646957 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" event={"ID":"9b869364-017a-4cdd-8501-c68fcee1d73b","Type":"ContainerStarted","Data":"ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea"} Nov 02 15:32:06.647197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.647051 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" event={"ID":"9b869364-017a-4cdd-8501-c68fcee1d73b","Type":"ContainerStarted","Data":"bb41fddf49dd6d37dc4744c6dc836ee1ec337aa34dae29ef41e6994fafb6e108"} Nov 02 15:32:06.647263 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.647192 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:32:06.648705 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.648683 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" event={"ID":"b785048a-948c-4334-a2cb-c12ff8990d24","Type":"ContainerStarted","Data":"147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0"} Nov 02 15:32:06.648801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.648719 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" event={"ID":"b785048a-948c-4334-a2cb-c12ff8990d24","Type":"ContainerStarted","Data":"b9d262174dd27d380e6bfbbeeebe4e331762bce6eae4f2288d2014e05743a4c7"} Nov 02 15:32:06.648801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.648765 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:32:06.758543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.758446 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" podStartSLOduration=1.758432074 podStartE2EDuration="1.758432074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:32:06.755916136 +0000 UTC m=+2028.958794194" watchObservedRunningTime="2026-11-02 15:32:06.758432074 +0000 UTC m=+2028.961310175" Nov 02 15:32:06.877294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:06.869064 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" podStartSLOduration=1.869046869 podStartE2EDuration="1.869046869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:32:06.864356259 +0000 UTC m=+2029.067234319" watchObservedRunningTime="2026-11-02 15:32:06.869046869 +0000 UTC m=+2029.071924930" Nov 02 15:32:07.502402 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:07.502366 5476 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:32:07.502755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:07.502415 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:32:09.671881 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:09.671850 5476 generic.go:356] "Generic (PLEG): container finished" podID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerID="9c3b0f25541ffecea7e8a236f2deb2ffb6642dd141c86d3012d25a0ea9a7c09f" exitCode=0 Nov 02 15:32:09.672399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:09.671922 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" event={"ID":"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a","Type":"ContainerDied","Data":"9c3b0f25541ffecea7e8a236f2deb2ffb6642dd141c86d3012d25a0ea9a7c09f"} Nov 02 15:32:10.119622 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.119599 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:32:10.236847 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.236778 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-envoy\") pod \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " Nov 02 15:32:10.236847 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.236812 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-certs\") pod \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " Nov 02 15:32:10.237127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.236858 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-socket\") pod \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " Nov 02 15:32:10.237127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.236896 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-kube-api-access-49sld\" (UniqueName: \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-kube-api-access-49sld\") pod \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " Nov 02 15:32:10.237127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.236935 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-token\" (UniqueName: \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-token\") pod \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " Nov 02 15:32:10.237127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.237044 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-podinfo\") pod \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " Nov 02 15:32:10.237342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.237110 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-certs" pod "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" (UID: "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:10.237342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.237139 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-data\") pod \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " Nov 02 15:32:10.237342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.237193 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istiod-ca-cert\") pod \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " Nov 02 15:32:10.237342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.237212 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-socket" pod "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" (UID: "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:10.237342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.237310 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-credential-socket\") pod \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\" (UID: \"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a\") " Nov 02 15:32:10.237623 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.237408 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-data" pod "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" (UID: "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:10.237623 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.237590 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-credential-socket" pod "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" (UID: "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:10.237735 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.237617 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istiod-ca-cert" pod "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" (UID: "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:10.238091 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.238063 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:10.238227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.238093 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:10.238227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.238110 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:10.238227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.238125 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:10.238227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.238139 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:10.239311 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.239284 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-token" pod "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" (UID: "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:10.239311 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.239290 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-podinfo" pod "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" (UID: "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:32:10.239456 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.239315 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-kube-api-access-49sld" pod "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" (UID: "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a"). InnerVolumeSpecName "kube-api-access-49sld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:10.239456 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.239407 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-envoy" pod "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" (UID: "b71c0212-6bf1-43ed-9d89-9dddc4cdba4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:10.339639 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.339600 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:10.339639 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.339639 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:10.339832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.339656 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49sld\" (UniqueName: \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-kube-api-access-49sld\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:10.339832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.339673 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:10.681810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.681765 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" event={"ID":"b71c0212-6bf1-43ed-9d89-9dddc4cdba4a","Type":"ContainerDied","Data":"116e38d86b5bfdfee78d2aea2b4161eae2add216f973a4a8ca676486ce7f6d8f"} Nov 02 15:32:10.682619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.681868 5476 scope.go:122] "RemoveContainer" containerID="9c3b0f25541ffecea7e8a236f2deb2ffb6642dd141c86d3012d25a0ea9a7c09f" Nov 02 15:32:10.682619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.682079 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9" Nov 02 15:32:10.883678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.883645 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9"] Nov 02 15:32:10.890775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:10.890751 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-qhrp9"] Nov 02 15:32:11.927071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:11.927031 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" path="/var/lib/kubelet/pods/b71c0212-6bf1-43ed-9d89-9dddc4cdba4a/volumes" Nov 02 15:32:17.658576 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:17.658547 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:32:17.659017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:17.658617 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:32:20.147313 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:32:20.147269 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:32:24.054299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:24.054261 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd"] Nov 02 15:32:24.101118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:24.101082 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29443935-t6vdd"] Nov 02 15:32:25.928832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:32:25.928798 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccea56fd-e01e-4948-b2a5-1bc7d0c263bd" path="/var/lib/kubelet/pods/ccea56fd-e01e-4948-b2a5-1bc7d0c263bd/volumes" Nov 02 15:32:50.147315 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:32:50.147277 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:33:10.098912 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:10.098873 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:33:18.507446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:18.507409 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:33:18.507861 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:18.507536 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:33:19.174543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:19.174503 5476 scope.go:122] "RemoveContainer" containerID="d6d3d059764d126c0a92a0b3454914bbdbe9d851ce758c5a1f76865b1b5ea5be" Nov 02 15:33:20.147087 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:33:20.147050 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:33:20.375313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:20.375284 5476 generic.go:356] "Generic (PLEG): container finished" podID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerID="f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878" exitCode=0 Nov 02 15:33:20.375497 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:20.375352 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" event={"ID":"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87","Type":"ContainerDied","Data":"f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878"} Nov 02 15:33:20.375822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:20.375806 5476 scope.go:122] "RemoveContainer" containerID="f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878" Nov 02 15:33:20.377613 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:20.377579 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:33:20.377703 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:20.377662 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:33:20.377746 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:20.377699 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:33:21.407689 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:21.407656 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" event={"ID":"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87","Type":"ContainerStarted","Data":"a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88"} Nov 02 15:33:22.043439 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:22.043406 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:33:22.043643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:22.043621 5476 patch_prober.go:29] interesting pod/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 15:33:22.043690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:22.043674 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 15:33:23.043859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:23.043822 5476 patch_prober.go:29] interesting pod/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 15:33:23.044299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:23.043884 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 15:33:24.043656 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:24.043622 5476 patch_prober.go:29] interesting pod/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 15:33:24.043843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:24.043679 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 15:33:25.044349 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:25.044314 5476 patch_prober.go:29] interesting pod/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 15:33:25.044747 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:25.044373 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 15:33:26.044382 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:26.044346 5476 patch_prober.go:29] interesting pod/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 15:33:26.044782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:26.044402 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 15:33:27.048684 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:27.048652 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:33:27.049127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:27.049092 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:33:27.049683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:27.049661 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:33:31.002972 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.002934 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp"] Nov 02 15:33:31.003621 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.003605 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerName="istio-proxy" Nov 02 15:33:31.003686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.003622 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerName="istio-proxy" Nov 02 15:33:31.003739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.003730 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71c0212-6bf1-43ed-9d89-9dddc4cdba4a" containerName="istio-proxy" Nov 02 15:33:31.007619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.007595 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.056943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.056916 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-5afebc-serving-cert\"" Nov 02 15:33:31.083585 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.083555 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.083737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.083622 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.083788 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.083735 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ab0548-fe2d-4034-8516-f2c70de24a5d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.083788 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.083771 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfr2k\" (UniqueName: \"kubernetes.io/projected/36ab0548-fe2d-4034-8516-f2c70de24a5d-kube-api-access-rfr2k\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.104379 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.104352 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp"] Nov 02 15:33:31.185126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.185092 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.185293 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.185202 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ab0548-fe2d-4034-8516-f2c70de24a5d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.185293 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.185243 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfr2k\" (UniqueName: \"kubernetes.io/projected/36ab0548-fe2d-4034-8516-f2c70de24a5d-kube-api-access-rfr2k\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.185427 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.185410 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.185568 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:33:31.185528 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-serving-cert: secret "maas-api-e2e-shared-b-w4-5afebc-serving-cert" not found Nov 02 15:33:31.185649 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:33:31.185611 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-maas-api-tls podName:36ab0548-fe2d-4034-8516-f2c70de24a5d nodeName:}" failed. No retries permitted until 2026-11-02 15:33:31.685592008 +0000 UTC m=+2113.888470060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-maas-api-tls") pod "maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" (UID: "36ab0548-fe2d-4034-8516-f2c70de24a5d") : secret "maas-api-e2e-shared-b-w4-5afebc-serving-cert" not found Nov 02 15:33:31.185887 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.185865 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ab0548-fe2d-4034-8516-f2c70de24a5d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.187502 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.187483 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.244441 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.244411 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfr2k\" (UniqueName: \"kubernetes.io/projected/36ab0548-fe2d-4034-8516-f2c70de24a5d-kube-api-access-rfr2k\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.693094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.693061 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.695564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.695527 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:31.924497 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:31.924467 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:32.302412 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:32.302313 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp"] Nov 02 15:33:32.307211 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:33:32.307177 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ab0548_fe2d_4034_8516_f2c70de24a5d.slice/crio-6b2ed8af5d7aac047f25093819f575432dddc1b0c3644e0c7b074ccce0b81245 WatchSource:0}: Error finding container 6b2ed8af5d7aac047f25093819f575432dddc1b0c3644e0c7b074ccce0b81245: Status 404 returned error can't find the container with id 6b2ed8af5d7aac047f25093819f575432dddc1b0c3644e0c7b074ccce0b81245 Nov 02 15:33:32.491221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:32.491180 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" event={"ID":"36ab0548-fe2d-4034-8516-f2c70de24a5d","Type":"ContainerStarted","Data":"6b2ed8af5d7aac047f25093819f575432dddc1b0c3644e0c7b074ccce0b81245"} Nov 02 15:33:33.502037 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:33.501935 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" event={"ID":"36ab0548-fe2d-4034-8516-f2c70de24a5d","Type":"ContainerStarted","Data":"dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c"} Nov 02 15:33:33.502037 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:33.502023 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:33.616173 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:33.616124 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" podStartSLOduration=2.7163477240000002 podStartE2EDuration="3.616110589s" totalImagesPullingTime="899.762865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:33:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:33:33.612958987 +0000 UTC m=+2115.815837047" watchObservedRunningTime="2026-11-02 15:33:33.616110589 +0000 UTC m=+2115.818988643" Nov 02 15:33:39.517619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:33:39.517590 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:33:50.146982 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:33:50.146937 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:34:01.571549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:01.571511 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:34:03.660907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:03.660862 5476 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:03.673112 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:03.673071 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:03.674194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:03.674161 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:03.713163 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:03.713119 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" probeResult="failure" output="Get \"https://10.134.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:03.722431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:03.722388 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:03.722779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:03.722659 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:03.842025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:03.837226 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:34:03.926166 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:03.926076 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 15:34:03.937216 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:03.937180 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Nov 02 15:34:05.512050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:05.497320 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x"] Nov 02 15:34:05.512050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:05.511716 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" podUID="90d7d3e9-465f-4da2-9ad8-4702fe5f9af5" containerName="maas-api" containerID="cri-o://e95ffcec1c67bff026fc69531fa0440dd8af519f974176335497489d2507ea22" gracePeriod=30 Nov 02 15:34:06.121663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.121626 5476 generic.go:356] "Generic (PLEG): container finished" podID="90d7d3e9-465f-4da2-9ad8-4702fe5f9af5" containerID="e95ffcec1c67bff026fc69531fa0440dd8af519f974176335497489d2507ea22" exitCode=0 Nov 02 15:34:06.132613 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.121829 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" event={"ID":"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5","Type":"ContainerDied","Data":"e95ffcec1c67bff026fc69531fa0440dd8af519f974176335497489d2507ea22"} Nov 02 15:34:06.132891 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.123778 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:34:06.311475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.311382 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-metrics-certs\") pod \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " Nov 02 15:34:06.311475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.311431 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-openshift-service-ca\") pod \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " Nov 02 15:34:06.311713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.311485 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-maas-api-tls\") pod \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " Nov 02 15:34:06.311713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.311541 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-kube-api-access-pm6ft\" (UniqueName: \"kubernetes.io/projected/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-kube-api-access-pm6ft\") pod \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\" (UID: \"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5\") " Nov 02 15:34:06.312115 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.312087 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-openshift-service-ca" pod "90d7d3e9-465f-4da2-9ad8-4702fe5f9af5" (UID: "90d7d3e9-465f-4da2-9ad8-4702fe5f9af5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:06.314221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.314160 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-maas-api-tls" pod "90d7d3e9-465f-4da2-9ad8-4702fe5f9af5" (UID: "90d7d3e9-465f-4da2-9ad8-4702fe5f9af5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:34:06.314358 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.314264 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-metrics-certs" pod "90d7d3e9-465f-4da2-9ad8-4702fe5f9af5" (UID: "90d7d3e9-465f-4da2-9ad8-4702fe5f9af5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:34:06.315873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.315837 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-kube-api-access-pm6ft" pod "90d7d3e9-465f-4da2-9ad8-4702fe5f9af5" (UID: "90d7d3e9-465f-4da2-9ad8-4702fe5f9af5"). InnerVolumeSpecName "kube-api-access-pm6ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:06.413662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.413628 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:06.413662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.413656 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:06.413662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.413667 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:06.413888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:06.413676 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm6ft\" (UniqueName: \"kubernetes.io/projected/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5-kube-api-access-pm6ft\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:07.140610 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:07.140578 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" Nov 02 15:34:07.141171 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:07.140617 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x" event={"ID":"90d7d3e9-465f-4da2-9ad8-4702fe5f9af5","Type":"ContainerDied","Data":"a84ba92b8f959cb71144f80eabc02f70ac586036ca733f5bcd4d5e36d4344e16"} Nov 02 15:34:07.141171 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:07.140689 5476 scope.go:122] "RemoveContainer" containerID="e95ffcec1c67bff026fc69531fa0440dd8af519f974176335497489d2507ea22" Nov 02 15:34:07.308249 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:07.308221 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x"] Nov 02 15:34:07.344952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:07.344927 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de358-cbb55bfb8-x262x"] Nov 02 15:34:07.924700 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:07.924667 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d7d3e9-465f-4da2-9ad8-4702fe5f9af5" path="/var/lib/kubelet/pods/90d7d3e9-465f-4da2-9ad8-4702fe5f9af5/volumes" Nov 02 15:34:14.143017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.142952 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj"] Nov 02 15:34:14.144608 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.144572 5476 cpu_manager.go:414] "containerMap: removing container" podUID="90d7d3e9-465f-4da2-9ad8-4702fe5f9af5" containerName="maas-api" Nov 02 15:34:14.144608 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.144603 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d7d3e9-465f-4da2-9ad8-4702fe5f9af5" containerName="maas-api" Nov 02 15:34:14.144809 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.144796 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d7d3e9-465f-4da2-9ad8-4702fe5f9af5" containerName="maas-api" Nov 02 15:34:14.149281 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.149241 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:34:14.226511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.222605 5476 generic.go:356] "Generic (PLEG): container finished" podID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerID="4bc155a3dd402aab33edb9b7de65971f3b12072da0087f6a331c1a690c434a65" exitCode=0 Nov 02 15:34:14.226511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.222658 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" event={"ID":"a2042e87-2978-48d1-a54f-a6c785cc68dd","Type":"ContainerDied","Data":"4bc155a3dd402aab33edb9b7de65971f3b12072da0087f6a331c1a690c434a65"} Nov 02 15:34:14.226511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.223191 5476 scope.go:122] "RemoveContainer" containerID="4bc155a3dd402aab33edb9b7de65971f3b12072da0087f6a331c1a690c434a65" Nov 02 15:34:14.226511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.225446 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:14.226511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.225537 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:14.226511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.225586 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:14.238133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.238091 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739b627b-2186-4b9a-bf9c-dd21c6c192b5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj\" (UID: \"739b627b-2186-4b9a-bf9c-dd21c6c192b5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:34:14.238450 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.238427 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnrc6\" (UniqueName: \"kubernetes.io/projected/739b627b-2186-4b9a-bf9c-dd21c6c192b5-kube-api-access-xnrc6\") pod \"payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj\" (UID: \"739b627b-2186-4b9a-bf9c-dd21c6c192b5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:34:14.257300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.257266 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-aa5832\"" Nov 02 15:34:14.257467 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.257381 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-aa5832-dockercfg-mcldg\"" Nov 02 15:34:14.292668 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.292631 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj"] Nov 02 15:34:14.340349 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.340318 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739b627b-2186-4b9a-bf9c-dd21c6c192b5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj\" (UID: \"739b627b-2186-4b9a-bf9c-dd21c6c192b5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:34:14.340653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.340624 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnrc6\" (UniqueName: \"kubernetes.io/projected/739b627b-2186-4b9a-bf9c-dd21c6c192b5-kube-api-access-xnrc6\") pod \"payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj\" (UID: \"739b627b-2186-4b9a-bf9c-dd21c6c192b5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:34:14.341063 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.341046 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739b627b-2186-4b9a-bf9c-dd21c6c192b5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj\" (UID: \"739b627b-2186-4b9a-bf9c-dd21c6c192b5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:34:14.398012 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.397926 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnrc6\" (UniqueName: \"kubernetes.io/projected/739b627b-2186-4b9a-bf9c-dd21c6c192b5-kube-api-access-xnrc6\") pod \"payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj\" (UID: \"739b627b-2186-4b9a-bf9c-dd21c6c192b5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:34:14.476538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.476505 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:34:14.859562 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:14.859536 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj"] Nov 02 15:34:14.862517 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:34:14.862482 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739b627b_2186_4b9a_bf9c_dd21c6c192b5.slice/crio-0ca3ce9904452f0ae1f2789d45c64b430559154fc24902cddd576a08b6dfa191 WatchSource:0}: Error finding container 0ca3ce9904452f0ae1f2789d45c64b430559154fc24902cddd576a08b6dfa191: Status 404 returned error can't find the container with id 0ca3ce9904452f0ae1f2789d45c64b430559154fc24902cddd576a08b6dfa191 Nov 02 15:34:15.231208 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:15.231105 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" event={"ID":"739b627b-2186-4b9a-bf9c-dd21c6c192b5","Type":"ContainerStarted","Data":"40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7"} Nov 02 15:34:15.231208 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:15.231163 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" event={"ID":"739b627b-2186-4b9a-bf9c-dd21c6c192b5","Type":"ContainerStarted","Data":"0ca3ce9904452f0ae1f2789d45c64b430559154fc24902cddd576a08b6dfa191"} Nov 02 15:34:15.231208 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:15.231203 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:34:15.233313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:15.233286 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" event={"ID":"a2042e87-2978-48d1-a54f-a6c785cc68dd","Type":"ContainerStarted","Data":"d9f8f706203cf777dd982dd33b3b81272f15f11b8da7229d16e448f3852383b0"} Nov 02 15:34:15.352197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:15.352142 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" podStartSLOduration=1.352126337 podStartE2EDuration="1.352126337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:15.350788764 +0000 UTC m=+2157.553666824" watchObservedRunningTime="2026-11-02 15:34:15.352126337 +0000 UTC m=+2157.555004397" Nov 02 15:34:15.420512 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:15.420465 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:34:15.543425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:15.543334 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:34:15.544990 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:15.544959 5476 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 15:34:15.545133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:15.545037 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 15:34:16.543526 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:16.543490 5476 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 15:34:16.543923 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:16.543545 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 15:34:16.915980 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:16.915941 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw"] Nov 02 15:34:16.916199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:16.916178 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" containerID="cri-o://0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8" gracePeriod=30 Nov 02 15:34:17.158959 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:17.158923 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr"] Nov 02 15:34:17.159277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:17.159217 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" containerID="cri-o://dc1c072823c9a0254e2091353f7483d2e35c5f96809c1ffaeb46fc4bfc79c071" gracePeriod=30 Nov 02 15:34:17.543367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:17.543330 5476 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 15:34:17.543579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:17.543394 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 15:34:18.543849 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:18.543817 5476 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 15:34:18.544292 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:18.543873 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 15:34:19.204511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:19.204481 5476 scope.go:122] "RemoveContainer" containerID="a1e2792ae7344fd6c30404f62609b06352ad43eb7496212371cb55954f2feddb" Nov 02 15:34:19.543580 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:19.543499 5476 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 02 15:34:19.543723 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:19.543578 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 02 15:34:20.149338 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:34:20.149300 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:34:20.665064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:20.665026 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:34:20.667763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:20.667726 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:34:20.912967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:20.912927 5476 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:34:20.913144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:20.913010 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:34:22.310611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.310583 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:34:22.319670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.319641 5476 generic.go:356] "Generic (PLEG): container finished" podID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerID="0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8" exitCode=0 Nov 02 15:34:22.319817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.319708 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" Nov 02 15:34:22.319817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.319734 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" event={"ID":"ea98dd01-bd7f-4d6e-8988-1c5f70959565","Type":"ContainerDied","Data":"0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8"} Nov 02 15:34:22.319817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.319798 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw" event={"ID":"ea98dd01-bd7f-4d6e-8988-1c5f70959565","Type":"ContainerDied","Data":"9f886c569db49e0c01cb8726d4510135a61cb588297aa868fe205dd6299dfc69"} Nov 02 15:34:22.320003 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.319826 5476 scope.go:122] "RemoveContainer" containerID="0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8" Nov 02 15:34:22.321884 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.321865 5476 generic.go:356] "Generic (PLEG): container finished" podID="9af3940e-fa13-4569-9951-a338c215ba0d" containerID="dc1c072823c9a0254e2091353f7483d2e35c5f96809c1ffaeb46fc4bfc79c071" exitCode=0 Nov 02 15:34:22.321989 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.321931 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" event={"ID":"9af3940e-fa13-4569-9951-a338c215ba0d","Type":"ContainerDied","Data":"dc1c072823c9a0254e2091353f7483d2e35c5f96809c1ffaeb46fc4bfc79c071"} Nov 02 15:34:22.334962 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.334941 5476 scope.go:122] "RemoveContainer" containerID="0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8" Nov 02 15:34:22.335301 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:34:22.335282 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8\": container with ID starting with 0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8 not found: ID does not exist" containerID="0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8" Nov 02 15:34:22.335348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.335313 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8"} err="failed to get container status \"0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8\": rpc error: code = NotFound desc = could not find container \"0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8\": container with ID starting with 0dfd88edb25f576b60d2e3dfe65915f25e7eace0afba60c4398efc62dd45eac8 not found: ID does not exist" Nov 02 15:34:22.335348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.335331 5476 scope.go:122] "RemoveContainer" containerID="5d57848a937bff863cfed4a1e34767560641123b211c2c3c612c21211555d498" Nov 02 15:34:22.399183 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.399146 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istiod-ca-cert\") pod \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " Nov 02 15:34:22.399385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.399216 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-kube-api-access-8rzxq\" (UniqueName: \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-kube-api-access-8rzxq\") pod \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " Nov 02 15:34:22.399385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.399275 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-data\") pod \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " Nov 02 15:34:22.399385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.399317 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-certs\") pod \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " Nov 02 15:34:22.399385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.399368 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-envoy\") pod \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " Nov 02 15:34:22.399677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.399401 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-podinfo\") pod \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " Nov 02 15:34:22.399677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.399428 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-credential-socket\") pod \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " Nov 02 15:34:22.399677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.399485 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-token\" (UniqueName: \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-token\") pod \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " Nov 02 15:34:22.399677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.399502 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-socket\") pod \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\" (UID: \"ea98dd01-bd7f-4d6e-8988-1c5f70959565\") " Nov 02 15:34:22.400379 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.400325 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-certs" pod "ea98dd01-bd7f-4d6e-8988-1c5f70959565" (UID: "ea98dd01-bd7f-4d6e-8988-1c5f70959565"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:22.400958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.400590 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-data" pod "ea98dd01-bd7f-4d6e-8988-1c5f70959565" (UID: "ea98dd01-bd7f-4d6e-8988-1c5f70959565"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:22.400958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.400645 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-socket" pod "ea98dd01-bd7f-4d6e-8988-1c5f70959565" (UID: "ea98dd01-bd7f-4d6e-8988-1c5f70959565"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:22.400958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.400880 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-credential-socket" pod "ea98dd01-bd7f-4d6e-8988-1c5f70959565" (UID: "ea98dd01-bd7f-4d6e-8988-1c5f70959565"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:22.400958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.400912 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istiod-ca-cert" pod "ea98dd01-bd7f-4d6e-8988-1c5f70959565" (UID: "ea98dd01-bd7f-4d6e-8988-1c5f70959565"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:22.403525 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.403470 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-kube-api-access-8rzxq" pod "ea98dd01-bd7f-4d6e-8988-1c5f70959565" (UID: "ea98dd01-bd7f-4d6e-8988-1c5f70959565"). InnerVolumeSpecName "kube-api-access-8rzxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:22.404150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.404123 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-envoy" pod "ea98dd01-bd7f-4d6e-8988-1c5f70959565" (UID: "ea98dd01-bd7f-4d6e-8988-1c5f70959565"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:22.404367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.404347 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-token" pod "ea98dd01-bd7f-4d6e-8988-1c5f70959565" (UID: "ea98dd01-bd7f-4d6e-8988-1c5f70959565"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:22.405150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.405098 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-podinfo" pod "ea98dd01-bd7f-4d6e-8988-1c5f70959565" (UID: "ea98dd01-bd7f-4d6e-8988-1c5f70959565"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:34:22.446858 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.446791 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:34:22.501465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.501432 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.501465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.501459 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.501465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.501469 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.501710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.501479 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rzxq\" (UniqueName: \"kubernetes.io/projected/ea98dd01-bd7f-4d6e-8988-1c5f70959565-kube-api-access-8rzxq\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.501710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.501490 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.501710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.501498 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.501710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.501508 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.501710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.501515 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea98dd01-bd7f-4d6e-8988-1c5f70959565-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.501710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.501524 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea98dd01-bd7f-4d6e-8988-1c5f70959565-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.602946 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.602915 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-envoy\") pod \"9af3940e-fa13-4569-9951-a338c215ba0d\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " Nov 02 15:34:22.603155 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.602984 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af3940e-fa13-4569-9951-a338c215ba0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af3940e-fa13-4569-9951-a338c215ba0d-istiod-ca-cert\") pod \"9af3940e-fa13-4569-9951-a338c215ba0d\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " Nov 02 15:34:22.603155 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.603078 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-kube-api-access-fkbh6\" (UniqueName: \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-kube-api-access-fkbh6\") pod \"9af3940e-fa13-4569-9951-a338c215ba0d\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " Nov 02 15:34:22.603484 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.603115 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-socket\") pod \"9af3940e-fa13-4569-9951-a338c215ba0d\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " Nov 02 15:34:22.603699 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.603452 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af3940e-fa13-4569-9951-a338c215ba0d-istiod-ca-cert" pod "9af3940e-fa13-4569-9951-a338c215ba0d" (UID: "9af3940e-fa13-4569-9951-a338c215ba0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:22.603699 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.603609 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-socket" pod "9af3940e-fa13-4569-9951-a338c215ba0d" (UID: "9af3940e-fa13-4569-9951-a338c215ba0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:22.603699 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.603529 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-credential-socket\") pod \"9af3940e-fa13-4569-9951-a338c215ba0d\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " Nov 02 15:34:22.603908 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.603726 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-data\") pod \"9af3940e-fa13-4569-9951-a338c215ba0d\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " Nov 02 15:34:22.603908 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.603759 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-certs\") pod \"9af3940e-fa13-4569-9951-a338c215ba0d\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " Nov 02 15:34:22.604038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.603968 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9af3940e-fa13-4569-9951-a338c215ba0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af3940e-fa13-4569-9951-a338c215ba0d-istio-podinfo\") pod \"9af3940e-fa13-4569-9951-a338c215ba0d\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " Nov 02 15:34:22.604038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.603959 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-data" pod "9af3940e-fa13-4569-9951-a338c215ba0d" (UID: "9af3940e-fa13-4569-9951-a338c215ba0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:22.604144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.604037 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-istio-token\" (UniqueName: \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-istio-token\") pod \"9af3940e-fa13-4569-9951-a338c215ba0d\" (UID: \"9af3940e-fa13-4569-9951-a338c215ba0d\") " Nov 02 15:34:22.604144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.604040 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-certs" pod "9af3940e-fa13-4569-9951-a338c215ba0d" (UID: "9af3940e-fa13-4569-9951-a338c215ba0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:22.604144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.604066 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-credential-socket" pod "9af3940e-fa13-4569-9951-a338c215ba0d" (UID: "9af3940e-fa13-4569-9951-a338c215ba0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:22.604752 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.604726 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.604854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.604757 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.604854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.604772 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.604854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.604787 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.604854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.604801 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af3940e-fa13-4569-9951-a338c215ba0d-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.607389 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.607363 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-istio-token" pod "9af3940e-fa13-4569-9951-a338c215ba0d" (UID: "9af3940e-fa13-4569-9951-a338c215ba0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:22.607481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.607364 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-envoy" pod "9af3940e-fa13-4569-9951-a338c215ba0d" (UID: "9af3940e-fa13-4569-9951-a338c215ba0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:22.607481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.607405 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-kube-api-access-fkbh6" pod "9af3940e-fa13-4569-9951-a338c215ba0d" (UID: "9af3940e-fa13-4569-9951-a338c215ba0d"). InnerVolumeSpecName "kube-api-access-fkbh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:22.608518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.608495 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9af3940e-fa13-4569-9951-a338c215ba0d-istio-podinfo" pod "9af3940e-fa13-4569-9951-a338c215ba0d" (UID: "9af3940e-fa13-4569-9951-a338c215ba0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:34:22.706446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.706382 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkbh6\" (UniqueName: \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-kube-api-access-fkbh6\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.706446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.706408 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af3940e-fa13-4569-9951-a338c215ba0d-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.706446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.706420 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af3940e-fa13-4569-9951-a338c215ba0d-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.706446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.706433 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af3940e-fa13-4569-9951-a338c215ba0d-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:22.740483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.740456 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw"] Nov 02 15:34:22.745051 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:22.745027 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-px2rw"] Nov 02 15:34:23.330988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:23.330955 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" Nov 02 15:34:23.331483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:23.331019 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr" event={"ID":"9af3940e-fa13-4569-9951-a338c215ba0d","Type":"ContainerDied","Data":"7606626f7268271a8a11f53516c4dc0ce7a326fe1038621a78c4e4683b2b5115"} Nov 02 15:34:23.331483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:23.331080 5476 scope.go:122] "RemoveContainer" containerID="dc1c072823c9a0254e2091353f7483d2e35c5f96809c1ffaeb46fc4bfc79c071" Nov 02 15:34:23.491739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:23.491710 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr"] Nov 02 15:34:23.537601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:23.537570 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0de358-openshift-default-7466f54655-gnmcr"] Nov 02 15:34:23.924248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:23.924218 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" path="/var/lib/kubelet/pods/9af3940e-fa13-4569-9951-a338c215ba0d/volumes" Nov 02 15:34:23.924682 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:23.924669 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" path="/var/lib/kubelet/pods/ea98dd01-bd7f-4d6e-8988-1c5f70959565/volumes" Nov 02 15:34:26.243544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:26.243515 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:34:28.528218 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.528133 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr"] Nov 02 15:34:28.528848 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.528830 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" Nov 02 15:34:28.528910 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.528849 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" Nov 02 15:34:28.528910 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.528862 5476 cpu_manager.go:414] "containerMap: removing container" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" Nov 02 15:34:28.528910 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.528867 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" Nov 02 15:34:28.529107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.528961 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea98dd01-bd7f-4d6e-8988-1c5f70959565" containerName="istio-proxy" Nov 02 15:34:28.529107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.528974 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" Nov 02 15:34:28.529107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.528984 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" Nov 02 15:34:28.533856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.533836 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:34:28.584677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.584648 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-822f77\"" Nov 02 15:34:28.589230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.589204 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr"] Nov 02 15:34:28.610552 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.610511 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da89a1aa-673c-4856-b66d-cf095a3d966b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr\" (UID: \"da89a1aa-673c-4856-b66d-cf095a3d966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:34:28.610747 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.610709 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2dzs\" (UniqueName: \"kubernetes.io/projected/da89a1aa-673c-4856-b66d-cf095a3d966b-kube-api-access-c2dzs\") pod \"payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr\" (UID: \"da89a1aa-673c-4856-b66d-cf095a3d966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:34:28.631437 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.631406 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-822f77-dockercfg-v8jtx\"" Nov 02 15:34:28.723271 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.723229 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2dzs\" (UniqueName: \"kubernetes.io/projected/da89a1aa-673c-4856-b66d-cf095a3d966b-kube-api-access-c2dzs\") pod \"payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr\" (UID: \"da89a1aa-673c-4856-b66d-cf095a3d966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:34:28.723481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.723446 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da89a1aa-673c-4856-b66d-cf095a3d966b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr\" (UID: \"da89a1aa-673c-4856-b66d-cf095a3d966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:34:28.744905 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.724592 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da89a1aa-673c-4856-b66d-cf095a3d966b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr\" (UID: \"da89a1aa-673c-4856-b66d-cf095a3d966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:34:28.745808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.745767 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr"] Nov 02 15:34:28.748944 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.748912 5476 cpu_manager.go:414] "containerMap: removing container" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" Nov 02 15:34:28.749128 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.749110 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af3940e-fa13-4569-9951-a338c215ba0d" containerName="istio-proxy" Nov 02 15:34:28.829138 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.822611 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2dzs\" (UniqueName: \"kubernetes.io/projected/da89a1aa-673c-4856-b66d-cf095a3d966b-kube-api-access-c2dzs\") pod \"payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr\" (UID: \"da89a1aa-673c-4856-b66d-cf095a3d966b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:34:28.874017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.847727 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:34:28.874017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.850385 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr"] Nov 02 15:34:28.874017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.850485 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:34:28.927510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.927224 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-822f77-766858fc97-692nr\" (UID: \"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:34:28.927510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:28.927304 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb4w2\" (UniqueName: \"kubernetes.io/projected/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-kube-api-access-xb4w2\") pod \"payload-processing-e2e-inf-a-822f77-766858fc97-692nr\" (UID: \"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:34:29.028654 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:29.028622 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-822f77-766858fc97-692nr\" (UID: \"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:34:29.028839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:29.028691 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb4w2\" (UniqueName: \"kubernetes.io/projected/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-kube-api-access-xb4w2\") pod \"payload-processing-e2e-inf-a-822f77-766858fc97-692nr\" (UID: \"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:34:29.029678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:29.029650 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-822f77-766858fc97-692nr\" (UID: \"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:34:29.086761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:29.086702 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb4w2\" (UniqueName: \"kubernetes.io/projected/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-kube-api-access-xb4w2\") pod \"payload-processing-e2e-inf-a-822f77-766858fc97-692nr\" (UID: \"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:34:29.246988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:29.246958 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:34:29.302897 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:29.302868 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr"] Nov 02 15:34:29.317350 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:34:29.307047 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda89a1aa_673c_4856_b66d_cf095a3d966b.slice/crio-b2694fee4732cc9ea6ec37d2b3608b5004ba855a25d9175624746f2e5cdb717b WatchSource:0}: Error finding container b2694fee4732cc9ea6ec37d2b3608b5004ba855a25d9175624746f2e5cdb717b: Status 404 returned error can't find the container with id b2694fee4732cc9ea6ec37d2b3608b5004ba855a25d9175624746f2e5cdb717b Nov 02 15:34:29.385568 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:29.385539 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" event={"ID":"da89a1aa-673c-4856-b66d-cf095a3d966b","Type":"ContainerStarted","Data":"b2694fee4732cc9ea6ec37d2b3608b5004ba855a25d9175624746f2e5cdb717b"} Nov 02 15:34:29.626439 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:29.626342 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr"] Nov 02 15:34:29.630666 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:34:29.630638 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4d8faa_f354_4fb8_9091_51ed0f1f29ff.slice/crio-fec95cb85004f6f39e26a76c062d91c78d76275e872fa99cd4d77365b5d0c2de WatchSource:0}: Error finding container fec95cb85004f6f39e26a76c062d91c78d76275e872fa99cd4d77365b5d0c2de: Status 404 returned error can't find the container with id fec95cb85004f6f39e26a76c062d91c78d76275e872fa99cd4d77365b5d0c2de Nov 02 15:34:30.396153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:30.396090 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" event={"ID":"da89a1aa-673c-4856-b66d-cf095a3d966b","Type":"ContainerStarted","Data":"56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c"} Nov 02 15:34:30.397214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:30.397179 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:34:30.399132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:30.399092 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" event={"ID":"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff","Type":"ContainerStarted","Data":"6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465"} Nov 02 15:34:30.399262 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:30.399146 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" event={"ID":"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff","Type":"ContainerStarted","Data":"fec95cb85004f6f39e26a76c062d91c78d76275e872fa99cd4d77365b5d0c2de"} Nov 02 15:34:30.418023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:30.399906 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:34:30.510391 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:30.510336 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" podStartSLOduration=2.510321223 podStartE2EDuration="2.510321223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:30.506210799 +0000 UTC m=+2172.709088861" watchObservedRunningTime="2026-11-02 15:34:30.510321223 +0000 UTC m=+2172.713199283" Nov 02 15:34:30.617886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:30.617828 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" podStartSLOduration=2.617810644 podStartE2EDuration="2.617810644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:30.613904021 +0000 UTC m=+2172.816782072" watchObservedRunningTime="2026-11-02 15:34:30.617810644 +0000 UTC m=+2172.820688704" Nov 02 15:34:33.574635 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:33.574588 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5"] Nov 02 15:34:33.575193 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:33.574824 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" containerID="cri-o://da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172" gracePeriod=30 Nov 02 15:34:34.618429 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.618382 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj"] Nov 02 15:34:34.623205 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.623181 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:34.701181 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.701143 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:34.701339 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.701305 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsb8k\" (UniqueName: \"kubernetes.io/projected/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-kube-api-access-zsb8k\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:34.701398 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.701371 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:34.720195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.720167 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:34:34.720351 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.720242 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:34:34.725511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.725487 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj"] Nov 02 15:34:34.803027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.802976 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:34.803190 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.803095 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsb8k\" (UniqueName: \"kubernetes.io/projected/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-kube-api-access-zsb8k\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:34.803190 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.803161 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:34.803626 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.803596 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:34.805464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.805443 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:34.861145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.861120 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsb8k\" (UniqueName: \"kubernetes.io/projected/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-kube-api-access-zsb8k\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:34.937403 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:34.937335 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:35.313433 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:35.313344 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj"] Nov 02 15:34:35.317779 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:34:35.317753 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4ed5ac_b2b6_4880_b219_26863a2d41f4.slice/crio-5655637016deffb5ca08fa2ccf13c7b1691278594dcce78da61e6eb2d326d6db WatchSource:0}: Error finding container 5655637016deffb5ca08fa2ccf13c7b1691278594dcce78da61e6eb2d326d6db: Status 404 returned error can't find the container with id 5655637016deffb5ca08fa2ccf13c7b1691278594dcce78da61e6eb2d326d6db Nov 02 15:34:35.436719 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:35.436684 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" event={"ID":"1c4ed5ac-b2b6-4880-b219-26863a2d41f4","Type":"ContainerStarted","Data":"5655637016deffb5ca08fa2ccf13c7b1691278594dcce78da61e6eb2d326d6db"} Nov 02 15:34:35.570430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:35.570340 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5"] Nov 02 15:34:35.570630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:35.570604 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" containerID="cri-o://380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b" gracePeriod=30 Nov 02 15:34:36.444927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:36.444842 5476 generic.go:356] "Generic (PLEG): container finished" podID="1c4ed5ac-b2b6-4880-b219-26863a2d41f4" containerID="8517e57ce9747b80e7ec258330edc2eee0f25535cdc57a21de18663dd271c7db" exitCode=0 Nov 02 15:34:36.444927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:36.444895 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" event={"ID":"1c4ed5ac-b2b6-4880-b219-26863a2d41f4","Type":"ContainerDied","Data":"8517e57ce9747b80e7ec258330edc2eee0f25535cdc57a21de18663dd271c7db"} Nov 02 15:34:37.624416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:37.624390 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:37.751876 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:37.751804 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-kube-api-access-zsb8k\" (UniqueName: \"kubernetes.io/projected/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-kube-api-access-zsb8k\") pod \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " Nov 02 15:34:37.752045 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:37.751960 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-openshift-ca\") pod \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " Nov 02 15:34:37.752045 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:37.752021 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-xks-ca\") pod \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\" (UID: \"1c4ed5ac-b2b6-4880-b219-26863a2d41f4\") " Nov 02 15:34:37.752311 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:37.752285 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-openshift-ca" pod "1c4ed5ac-b2b6-4880-b219-26863a2d41f4" (UID: "1c4ed5ac-b2b6-4880-b219-26863a2d41f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:37.752586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:37.752561 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:37.754244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:37.754222 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-xks-ca" pod "1c4ed5ac-b2b6-4880-b219-26863a2d41f4" (UID: "1c4ed5ac-b2b6-4880-b219-26863a2d41f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:34:37.754481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:37.754460 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-kube-api-access-zsb8k" pod "1c4ed5ac-b2b6-4880-b219-26863a2d41f4" (UID: "1c4ed5ac-b2b6-4880-b219-26863a2d41f4"). InnerVolumeSpecName "kube-api-access-zsb8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:37.853401 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:37.853370 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsb8k\" (UniqueName: \"kubernetes.io/projected/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-kube-api-access-zsb8k\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:37.853401 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:37.853396 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4ed5ac-b2b6-4880-b219-26863a2d41f4-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:38.461325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.461298 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" Nov 02 15:34:38.461543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.461336 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj" event={"ID":"1c4ed5ac-b2b6-4880-b219-26863a2d41f4","Type":"ContainerDied","Data":"5655637016deffb5ca08fa2ccf13c7b1691278594dcce78da61e6eb2d326d6db"} Nov 02 15:34:38.461543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.461385 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5655637016deffb5ca08fa2ccf13c7b1691278594dcce78da61e6eb2d326d6db" Nov 02 15:34:38.847948 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.847923 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:34:38.965654 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.965612 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-certs\") pod \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " Nov 02 15:34:38.965825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.965677 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-token\") pod \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " Nov 02 15:34:38.965825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.965707 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-credential-socket\") pod \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " Nov 02 15:34:38.965825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.965740 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-envoy\") pod \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " Nov 02 15:34:38.965825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.965772 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-data\") pod \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " Nov 02 15:34:38.965825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.965800 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-podinfo\") pod \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " Nov 02 15:34:38.966046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.965835 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-socket\") pod \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " Nov 02 15:34:38.966046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.965854 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-certs" pod "b7bf0ff2-c698-4896-a0bc-a32bab520c1a" (UID: "b7bf0ff2-c698-4896-a0bc-a32bab520c1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:38.966046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.965896 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-kube-api-access-8rprw\" (UniqueName: \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-kube-api-access-8rprw\") pod \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " Nov 02 15:34:38.966046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.965936 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istiod-ca-cert\") pod \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\" (UID: \"b7bf0ff2-c698-4896-a0bc-a32bab520c1a\") " Nov 02 15:34:38.966486 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.966340 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-credential-socket" pod "b7bf0ff2-c698-4896-a0bc-a32bab520c1a" (UID: "b7bf0ff2-c698-4896-a0bc-a32bab520c1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:38.966486 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.966459 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-data" pod "b7bf0ff2-c698-4896-a0bc-a32bab520c1a" (UID: "b7bf0ff2-c698-4896-a0bc-a32bab520c1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:38.966674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.966598 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:38.966674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.966608 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-socket" pod "b7bf0ff2-c698-4896-a0bc-a32bab520c1a" (UID: "b7bf0ff2-c698-4896-a0bc-a32bab520c1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:38.966674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.966619 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:38.966674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.966634 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:38.966674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.966654 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istiod-ca-cert" pod "b7bf0ff2-c698-4896-a0bc-a32bab520c1a" (UID: "b7bf0ff2-c698-4896-a0bc-a32bab520c1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:38.967936 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.967904 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-token" pod "b7bf0ff2-c698-4896-a0bc-a32bab520c1a" (UID: "b7bf0ff2-c698-4896-a0bc-a32bab520c1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:38.967936 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.967922 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-podinfo" pod "b7bf0ff2-c698-4896-a0bc-a32bab520c1a" (UID: "b7bf0ff2-c698-4896-a0bc-a32bab520c1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:34:38.968176 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.968159 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-envoy" pod "b7bf0ff2-c698-4896-a0bc-a32bab520c1a" (UID: "b7bf0ff2-c698-4896-a0bc-a32bab520c1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:38.968475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:38.968454 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-kube-api-access-8rprw" pod "b7bf0ff2-c698-4896-a0bc-a32bab520c1a" (UID: "b7bf0ff2-c698-4896-a0bc-a32bab520c1a"). InnerVolumeSpecName "kube-api-access-8rprw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:39.068536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.068459 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:39.068536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.068493 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rprw\" (UniqueName: \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-kube-api-access-8rprw\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:39.068536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.068507 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:39.068536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.068521 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:39.068536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.068536 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:39.068813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.068550 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bf0ff2-c698-4896-a0bc-a32bab520c1a-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:39.469799 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.469752 5476 generic.go:356] "Generic (PLEG): container finished" podID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerID="da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172" exitCode=0 Nov 02 15:34:39.469966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.469805 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" Nov 02 15:34:39.469966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.469823 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" event={"ID":"b7bf0ff2-c698-4896-a0bc-a32bab520c1a","Type":"ContainerDied","Data":"da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172"} Nov 02 15:34:39.469966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.469878 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5" event={"ID":"b7bf0ff2-c698-4896-a0bc-a32bab520c1a","Type":"ContainerDied","Data":"9a19206e2eeb2ef513db7ae95a60638c03bcf6ab26148b38584c6de1705caab8"} Nov 02 15:34:39.469966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.469902 5476 scope.go:122] "RemoveContainer" containerID="da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172" Nov 02 15:34:39.481416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.481396 5476 scope.go:122] "RemoveContainer" containerID="da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172" Nov 02 15:34:39.481656 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:34:39.481640 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172\": container with ID starting with da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172 not found: ID does not exist" containerID="da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172" Nov 02 15:34:39.481708 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.481663 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172"} err="failed to get container status \"da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172\": rpc error: code = NotFound desc = could not find container \"da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172\": container with ID starting with da41913ae458c03ff7a4af6ef06e439b9c6bcb57fceafac061cf215f62134172 not found: ID does not exist" Nov 02 15:34:39.629695 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.629659 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5"] Nov 02 15:34:39.670735 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.670704 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-sxjx5"] Nov 02 15:34:39.923609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:39.923574 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" path="/var/lib/kubelet/pods/b7bf0ff2-c698-4896-a0bc-a32bab520c1a/volumes" Nov 02 15:34:40.844351 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.844284 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:34:40.994960 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.994926 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-envoy\") pod \"ea558aff-5d34-4e17-9194-4cefdf5c266e\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " Nov 02 15:34:40.995452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995005 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-credential-socket\") pod \"ea558aff-5d34-4e17-9194-4cefdf5c266e\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " Nov 02 15:34:40.995452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995065 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-socket\") pod \"ea558aff-5d34-4e17-9194-4cefdf5c266e\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " Nov 02 15:34:40.995452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995119 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-token\" (UniqueName: \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-token\") pod \"ea558aff-5d34-4e17-9194-4cefdf5c266e\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " Nov 02 15:34:40.995452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995184 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-podinfo\") pod \"ea558aff-5d34-4e17-9194-4cefdf5c266e\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " Nov 02 15:34:40.995452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995256 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-data\") pod \"ea558aff-5d34-4e17-9194-4cefdf5c266e\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " Nov 02 15:34:40.995452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995306 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea558aff-5d34-4e17-9194-4cefdf5c266e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea558aff-5d34-4e17-9194-4cefdf5c266e-istiod-ca-cert\") pod \"ea558aff-5d34-4e17-9194-4cefdf5c266e\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " Nov 02 15:34:40.995452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995304 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-credential-socket" pod "ea558aff-5d34-4e17-9194-4cefdf5c266e" (UID: "ea558aff-5d34-4e17-9194-4cefdf5c266e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:40.995452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995385 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-kube-api-access-8md4c\" (UniqueName: \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-kube-api-access-8md4c\") pod \"ea558aff-5d34-4e17-9194-4cefdf5c266e\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " Nov 02 15:34:40.995452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995392 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-socket" pod "ea558aff-5d34-4e17-9194-4cefdf5c266e" (UID: "ea558aff-5d34-4e17-9194-4cefdf5c266e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:40.995452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995451 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-certs\") pod \"ea558aff-5d34-4e17-9194-4cefdf5c266e\" (UID: \"ea558aff-5d34-4e17-9194-4cefdf5c266e\") " Nov 02 15:34:40.996016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995638 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-data" pod "ea558aff-5d34-4e17-9194-4cefdf5c266e" (UID: "ea558aff-5d34-4e17-9194-4cefdf5c266e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:40.996016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995742 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea558aff-5d34-4e17-9194-4cefdf5c266e-istiod-ca-cert" pod "ea558aff-5d34-4e17-9194-4cefdf5c266e" (UID: "ea558aff-5d34-4e17-9194-4cefdf5c266e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:40.996139 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.995989 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-certs" pod "ea558aff-5d34-4e17-9194-4cefdf5c266e" (UID: "ea558aff-5d34-4e17-9194-4cefdf5c266e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:40.996443 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.996411 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:40.996570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.996447 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:40.996570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.996462 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:40.996570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.996476 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:40.996570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.996490 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea558aff-5d34-4e17-9194-4cefdf5c266e-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:40.997468 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.997440 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-envoy" pod "ea558aff-5d34-4e17-9194-4cefdf5c266e" (UID: "ea558aff-5d34-4e17-9194-4cefdf5c266e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:40.997563 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.997520 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-podinfo" pod "ea558aff-5d34-4e17-9194-4cefdf5c266e" (UID: "ea558aff-5d34-4e17-9194-4cefdf5c266e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:34:40.997859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.997841 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-kube-api-access-8md4c" pod "ea558aff-5d34-4e17-9194-4cefdf5c266e" (UID: "ea558aff-5d34-4e17-9194-4cefdf5c266e"). InnerVolumeSpecName "kube-api-access-8md4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:40.997919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:40.997854 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-token" pod "ea558aff-5d34-4e17-9194-4cefdf5c266e" (UID: "ea558aff-5d34-4e17-9194-4cefdf5c266e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:41.097692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.097617 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8md4c\" (UniqueName: \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-kube-api-access-8md4c\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:41.097692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.097644 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:41.097692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.097654 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:41.097692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.097663 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea558aff-5d34-4e17-9194-4cefdf5c266e-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:41.486067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.486036 5476 generic.go:356] "Generic (PLEG): container finished" podID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerID="380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b" exitCode=0 Nov 02 15:34:41.486243 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.486093 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" Nov 02 15:34:41.486243 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.486120 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" event={"ID":"ea558aff-5d34-4e17-9194-4cefdf5c266e","Type":"ContainerDied","Data":"380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b"} Nov 02 15:34:41.486243 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.486179 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5" event={"ID":"ea558aff-5d34-4e17-9194-4cefdf5c266e","Type":"ContainerDied","Data":"d917ffaeef14ffb3abd21fac0202dd8de21c61e85a5aedd25b3efcbab17b0be8"} Nov 02 15:34:41.486243 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.486208 5476 scope.go:122] "RemoveContainer" containerID="380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b" Nov 02 15:34:41.497274 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.497256 5476 scope.go:122] "RemoveContainer" containerID="380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b" Nov 02 15:34:41.497524 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:34:41.497504 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b\": container with ID starting with 380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b not found: ID does not exist" containerID="380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b" Nov 02 15:34:41.497572 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.497532 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b"} err="failed to get container status \"380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b\": rpc error: code = NotFound desc = could not find container \"380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b\": container with ID starting with 380ff74d656d34ce752b8751e6ad84bfd2903b0d4337eafe52e196e7174ee69b not found: ID does not exist" Nov 02 15:34:41.688228 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.688198 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5"] Nov 02 15:34:41.734632 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.734602 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-9zfc5"] Nov 02 15:34:41.923842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:41.923814 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" path="/var/lib/kubelet/pods/ea558aff-5d34-4e17-9194-4cefdf5c266e/volumes" Nov 02 15:34:42.415941 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:42.415917 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:34:42.416420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:42.415985 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:34:50.147675 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:34:50.147634 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:34:51.120874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:34:51.120844 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:35:00.750024 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.749972 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f"] Nov 02 15:35:00.750666 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.750649 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" Nov 02 15:35:00.750666 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.750667 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" Nov 02 15:35:00.750759 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.750680 5476 cpu_manager.go:414] "containerMap: removing container" podUID="1c4ed5ac-b2b6-4880-b219-26863a2d41f4" containerName="revoke-keys" Nov 02 15:35:00.750759 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.750687 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4ed5ac-b2b6-4880-b219-26863a2d41f4" containerName="revoke-keys" Nov 02 15:35:00.750759 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.750737 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" Nov 02 15:35:00.750759 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.750742 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" Nov 02 15:35:00.750887 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.750824 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7bf0ff2-c698-4896-a0bc-a32bab520c1a" containerName="istio-proxy" Nov 02 15:35:00.750887 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.750833 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4ed5ac-b2b6-4880-b219-26863a2d41f4" containerName="revoke-keys" Nov 02 15:35:00.750887 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.750850 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea558aff-5d34-4e17-9194-4cefdf5c266e" containerName="istio-proxy" Nov 02 15:35:00.755858 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.755843 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:00.850108 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.850071 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzgst\" (UniqueName: \"kubernetes.io/projected/5525cf5a-9372-47b7-b913-1e172d9235b4-kube-api-access-kzgst\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:00.850288 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.850250 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:00.850336 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.850322 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:00.852304 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.852285 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:35:00.852518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.852504 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:35:00.855852 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.855831 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f"] Nov 02 15:35:00.951490 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.951453 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:00.951690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.951558 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:00.951690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.951662 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzgst\" (UniqueName: \"kubernetes.io/projected/5525cf5a-9372-47b7-b913-1e172d9235b4-kube-api-access-kzgst\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:00.952240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.952217 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:00.953883 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:00.953852 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:01.011291 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:01.010312 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzgst\" (UniqueName: \"kubernetes.io/projected/5525cf5a-9372-47b7-b913-1e172d9235b4-kube-api-access-kzgst\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:01.069035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:01.068978 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:01.446818 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:01.446782 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f"] Nov 02 15:35:01.451958 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:35:01.451925 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5525cf5a_9372_47b7_b913_1e172d9235b4.slice/crio-de926ec0b286af661eb4a326353c0fa30b4337d4e86d0d063221dd39dbab3351 WatchSource:0}: Error finding container de926ec0b286af661eb4a326353c0fa30b4337d4e86d0d063221dd39dbab3351: Status 404 returned error can't find the container with id de926ec0b286af661eb4a326353c0fa30b4337d4e86d0d063221dd39dbab3351 Nov 02 15:35:01.649755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:01.649716 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" event={"ID":"5525cf5a-9372-47b7-b913-1e172d9235b4","Type":"ContainerStarted","Data":"de926ec0b286af661eb4a326353c0fa30b4337d4e86d0d063221dd39dbab3351"} Nov 02 15:35:02.660458 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:02.660418 5476 generic.go:356] "Generic (PLEG): container finished" podID="5525cf5a-9372-47b7-b913-1e172d9235b4" containerID="e30f0496023ded410a1bb27f3e1fa769b71c0e134e002c49e0793cd7218926e4" exitCode=0 Nov 02 15:35:02.660895 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:02.660473 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" event={"ID":"5525cf5a-9372-47b7-b913-1e172d9235b4","Type":"ContainerDied","Data":"e30f0496023ded410a1bb27f3e1fa769b71c0e134e002c49e0793cd7218926e4"} Nov 02 15:35:03.831034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:03.830984 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:03.994468 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:03.994393 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-openshift-ca\") pod \"5525cf5a-9372-47b7-b913-1e172d9235b4\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " Nov 02 15:35:03.994672 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:03.994652 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-xks-ca\") pod \"5525cf5a-9372-47b7-b913-1e172d9235b4\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " Nov 02 15:35:03.994776 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:03.994765 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5525cf5a-9372-47b7-b913-1e172d9235b4-kube-api-access-kzgst\" (UniqueName: \"kubernetes.io/projected/5525cf5a-9372-47b7-b913-1e172d9235b4-kube-api-access-kzgst\") pod \"5525cf5a-9372-47b7-b913-1e172d9235b4\" (UID: \"5525cf5a-9372-47b7-b913-1e172d9235b4\") " Nov 02 15:35:03.994819 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:03.994771 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-openshift-ca" pod "5525cf5a-9372-47b7-b913-1e172d9235b4" (UID: "5525cf5a-9372-47b7-b913-1e172d9235b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:03.995472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:03.995447 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:03.996821 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:03.996791 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5525cf5a-9372-47b7-b913-1e172d9235b4-kube-api-access-kzgst" pod "5525cf5a-9372-47b7-b913-1e172d9235b4" (UID: "5525cf5a-9372-47b7-b913-1e172d9235b4"). InnerVolumeSpecName "kube-api-access-kzgst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:03.996935 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:03.996852 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-xks-ca" pod "5525cf5a-9372-47b7-b913-1e172d9235b4" (UID: "5525cf5a-9372-47b7-b913-1e172d9235b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:35:04.096742 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:04.096717 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5525cf5a-9372-47b7-b913-1e172d9235b4-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:04.096742 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:04.096740 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzgst\" (UniqueName: \"kubernetes.io/projected/5525cf5a-9372-47b7-b913-1e172d9235b4-kube-api-access-kzgst\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:04.676774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:04.676734 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" event={"ID":"5525cf5a-9372-47b7-b913-1e172d9235b4","Type":"ContainerDied","Data":"de926ec0b286af661eb4a326353c0fa30b4337d4e86d0d063221dd39dbab3351"} Nov 02 15:35:04.676956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:04.676803 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de926ec0b286af661eb4a326353c0fa30b4337d4e86d0d063221dd39dbab3351" Nov 02 15:35:04.676956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:04.676748 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f" Nov 02 15:35:09.372156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.372123 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq"] Nov 02 15:35:09.372723 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.372710 5476 cpu_manager.go:414] "containerMap: removing container" podUID="5525cf5a-9372-47b7-b913-1e172d9235b4" containerName="revoke-keys" Nov 02 15:35:09.372772 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.372724 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5525cf5a-9372-47b7-b913-1e172d9235b4" containerName="revoke-keys" Nov 02 15:35:09.372857 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.372848 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5525cf5a-9372-47b7-b913-1e172d9235b4" containerName="revoke-keys" Nov 02 15:35:09.376691 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.376675 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.437904 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.437871 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq"] Nov 02 15:35:09.473005 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.472974 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2203c8-openshift-default-dockercfg-nd49d\"" Nov 02 15:35:09.477320 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.477292 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-podinfo\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.477492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.477358 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-certs\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.477564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.477493 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-data\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.477564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.477526 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-socket\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.477686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.477618 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-token\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.477686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.477649 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-envoy\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.477686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.477681 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-credential-socket\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.477852 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.477785 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nf86\" (UniqueName: \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-kube-api-access-7nf86\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.477852 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.477808 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istiod-ca-cert\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.579064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.578971 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-token\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.579064 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.579062 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-envoy\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.579341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.579118 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-credential-socket\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.579341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.579179 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nf86\" (UniqueName: \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-kube-api-access-7nf86\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.579341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.579232 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istiod-ca-cert\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.579510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.579367 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-podinfo\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.579510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.579448 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-certs\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.581544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.581486 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-certs\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.582433 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.582388 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-credential-socket\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.582888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.582862 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-data\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.583247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.582922 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-socket\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.584164 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.584066 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-data\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.584631 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.584588 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-podinfo\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.584928 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.584899 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-socket\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.585596 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.585564 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istiod-ca-cert\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.586352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.586294 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-envoy\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.640917 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.640840 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-token\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.644739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.644712 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nf86\" (UniqueName: \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-kube-api-access-7nf86\") pod \"e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:09.692055 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:09.692029 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:10.125477 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:10.125440 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq"] Nov 02 15:35:10.129874 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:35:10.129846 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf47ed4a_e6d2_4c9d_b29a_61830fb32bf5.slice/crio-8349424c9aa7ed66e023e2094c7352f3d9fc19ac47e7a8c665c7f36e53b95eaa WatchSource:0}: Error finding container 8349424c9aa7ed66e023e2094c7352f3d9fc19ac47e7a8c665c7f36e53b95eaa: Status 404 returned error can't find the container with id 8349424c9aa7ed66e023e2094c7352f3d9fc19ac47e7a8c665c7f36e53b95eaa Nov 02 15:35:10.132580 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:10.132548 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:35:10.132649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:10.132630 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:35:10.132693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:10.132669 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:35:10.728763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:10.728728 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" event={"ID":"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5","Type":"ContainerStarted","Data":"fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8"} Nov 02 15:35:10.729297 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:10.728793 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" event={"ID":"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5","Type":"ContainerStarted","Data":"8349424c9aa7ed66e023e2094c7352f3d9fc19ac47e7a8c665c7f36e53b95eaa"} Nov 02 15:35:10.844517 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:10.844464 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" podStartSLOduration=1.844451208 podStartE2EDuration="1.844451208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:35:10.842912998 +0000 UTC m=+2213.045791055" watchObservedRunningTime="2026-11-02 15:35:10.844451208 +0000 UTC m=+2213.047329268" Nov 02 15:35:11.692763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:11.692734 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:11.698396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:11.698373 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:11.745304 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:11.745269 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:11.746149 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:11.746126 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:35:17.485059 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:17.481433 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg"] Nov 02 15:35:17.489971 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:17.489930 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:35:17.501126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:17.501062 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd84d132-0deb-4c37-943d-1bb1a411fb41-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg\" (UID: \"bd84d132-0deb-4c37-943d-1bb1a411fb41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:35:17.501330 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:17.501315 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btv74\" (UniqueName: \"kubernetes.io/projected/bd84d132-0deb-4c37-943d-1bb1a411fb41-kube-api-access-btv74\") pod \"payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg\" (UID: \"bd84d132-0deb-4c37-943d-1bb1a411fb41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:35:17.567875 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:17.563794 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2203c8\"" Nov 02 15:35:17.576366 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:17.576334 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg"] Nov 02 15:35:17.605924 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:17.604086 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btv74\" (UniqueName: \"kubernetes.io/projected/bd84d132-0deb-4c37-943d-1bb1a411fb41-kube-api-access-btv74\") pod \"payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg\" (UID: \"bd84d132-0deb-4c37-943d-1bb1a411fb41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:35:17.605924 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:17.604202 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd84d132-0deb-4c37-943d-1bb1a411fb41-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg\" (UID: \"bd84d132-0deb-4c37-943d-1bb1a411fb41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:35:17.605924 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:17.605142 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd84d132-0deb-4c37-943d-1bb1a411fb41-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg\" (UID: \"bd84d132-0deb-4c37-943d-1bb1a411fb41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:35:17.661464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:17.661433 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btv74\" (UniqueName: \"kubernetes.io/projected/bd84d132-0deb-4c37-943d-1bb1a411fb41-kube-api-access-btv74\") pod \"payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg\" (UID: \"bd84d132-0deb-4c37-943d-1bb1a411fb41\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:35:17.852453 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:17.852380 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:35:18.240874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.240832 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg"] Nov 02 15:35:18.245521 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:35:18.245492 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd84d132_0deb_4c37_943d_1bb1a411fb41.slice/crio-220838774e7d52bf70e5a8ac6c14e0f89d070689592bac5a880d48fb65559309 WatchSource:0}: Error finding container 220838774e7d52bf70e5a8ac6c14e0f89d070689592bac5a880d48fb65559309: Status 404 returned error can't find the container with id 220838774e7d52bf70e5a8ac6c14e0f89d070689592bac5a880d48fb65559309 Nov 02 15:35:18.802102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.802064 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" event={"ID":"bd84d132-0deb-4c37-943d-1bb1a411fb41","Type":"ContainerStarted","Data":"084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8"} Nov 02 15:35:18.802534 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.802128 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" event={"ID":"bd84d132-0deb-4c37-943d-1bb1a411fb41","Type":"ContainerStarted","Data":"220838774e7d52bf70e5a8ac6c14e0f89d070689592bac5a880d48fb65559309"} Nov 02 15:35:18.802534 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.802271 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:35:18.903582 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.903552 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v"] Nov 02 15:35:18.907690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.907674 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:18.914507 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.914455 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" podStartSLOduration=1.914432137 podStartE2EDuration="1.914432137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:35:18.910594028 +0000 UTC m=+2221.113472088" watchObservedRunningTime="2026-11-02 15:35:18.914432137 +0000 UTC m=+2221.117310198" Nov 02 15:35:18.923532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.923508 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:18.923651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.923633 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckld\" (UniqueName: \"kubernetes.io/projected/1963f89a-b30d-4e51-abd7-b4407a986981-kube-api-access-2ckld\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:18.923738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.923713 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:18.959664 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.959628 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:35:18.960073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.960051 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:35:18.964244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:18.964222 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v"] Nov 02 15:35:19.025118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:19.025084 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckld\" (UniqueName: \"kubernetes.io/projected/1963f89a-b30d-4e51-abd7-b4407a986981-kube-api-access-2ckld\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:19.025685 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:19.025658 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:19.026391 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:19.026366 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:19.034049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:19.033958 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:19.047735 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:19.047704 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:19.085176 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:19.085109 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckld\" (UniqueName: \"kubernetes.io/projected/1963f89a-b30d-4e51-abd7-b4407a986981-kube-api-access-2ckld\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:19.223251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:19.223221 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:19.238689 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:19.238657 5476 scope.go:122] "RemoveContainer" containerID="faf33abf1997ef00bfc6fc7824f6a8ad6e397dfb2f5590e1eb766ec8b27973db" Nov 02 15:35:19.607520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:19.607489 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v"] Nov 02 15:35:19.611708 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:35:19.611676 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1963f89a_b30d_4e51_abd7_b4407a986981.slice/crio-7a2cbc9a8159498a05042ca84c584d7c56ebc307df4f1a6159c86193ae5958a7 WatchSource:0}: Error finding container 7a2cbc9a8159498a05042ca84c584d7c56ebc307df4f1a6159c86193ae5958a7: Status 404 returned error can't find the container with id 7a2cbc9a8159498a05042ca84c584d7c56ebc307df4f1a6159c86193ae5958a7 Nov 02 15:35:19.810830 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:19.810793 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" event={"ID":"1963f89a-b30d-4e51-abd7-b4407a986981","Type":"ContainerStarted","Data":"7a2cbc9a8159498a05042ca84c584d7c56ebc307df4f1a6159c86193ae5958a7"} Nov 02 15:35:20.147172 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:35:20.147081 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:35:20.820367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:20.820283 5476 generic.go:356] "Generic (PLEG): container finished" podID="1963f89a-b30d-4e51-abd7-b4407a986981" containerID="d09395bcb7005d3fa543fa07eb34e98a473e4538c67b91c8ba8941a3ff5cad93" exitCode=0 Nov 02 15:35:20.820733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:20.820364 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" event={"ID":"1963f89a-b30d-4e51-abd7-b4407a986981","Type":"ContainerDied","Data":"d09395bcb7005d3fa543fa07eb34e98a473e4538c67b91c8ba8941a3ff5cad93"} Nov 02 15:35:21.987985 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:21.987955 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:22.068684 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.068651 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-xks-ca\") pod \"1963f89a-b30d-4e51-abd7-b4407a986981\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " Nov 02 15:35:22.068866 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.068704 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1963f89a-b30d-4e51-abd7-b4407a986981-kube-api-access-2ckld\" (UniqueName: \"kubernetes.io/projected/1963f89a-b30d-4e51-abd7-b4407a986981-kube-api-access-2ckld\") pod \"1963f89a-b30d-4e51-abd7-b4407a986981\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " Nov 02 15:35:22.068944 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.068882 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-openshift-ca\") pod \"1963f89a-b30d-4e51-abd7-b4407a986981\" (UID: \"1963f89a-b30d-4e51-abd7-b4407a986981\") " Nov 02 15:35:22.069225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.069200 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-openshift-ca" pod "1963f89a-b30d-4e51-abd7-b4407a986981" (UID: "1963f89a-b30d-4e51-abd7-b4407a986981"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:22.069640 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.069617 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:22.070825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.070804 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-xks-ca" pod "1963f89a-b30d-4e51-abd7-b4407a986981" (UID: "1963f89a-b30d-4e51-abd7-b4407a986981"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:35:22.070921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.070900 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1963f89a-b30d-4e51-abd7-b4407a986981-kube-api-access-2ckld" pod "1963f89a-b30d-4e51-abd7-b4407a986981" (UID: "1963f89a-b30d-4e51-abd7-b4407a986981"). InnerVolumeSpecName "kube-api-access-2ckld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:22.170627 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.170593 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1963f89a-b30d-4e51-abd7-b4407a986981-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:22.170627 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.170622 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ckld\" (UniqueName: \"kubernetes.io/projected/1963f89a-b30d-4e51-abd7-b4407a986981-kube-api-access-2ckld\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:22.836044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.835987 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" Nov 02 15:35:22.836044 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.836026 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v" event={"ID":"1963f89a-b30d-4e51-abd7-b4407a986981","Type":"ContainerDied","Data":"7a2cbc9a8159498a05042ca84c584d7c56ebc307df4f1a6159c86193ae5958a7"} Nov 02 15:35:22.836256 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:22.836087 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a2cbc9a8159498a05042ca84c584d7c56ebc307df4f1a6159c86193ae5958a7" Nov 02 15:35:29.814530 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:29.814500 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:35:30.358069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:30.358038 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9"] Nov 02 15:35:34.250891 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:34.250859 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:35:35.358965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.358919 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" podUID="06a08d9f-a36e-4a36-90ef-414f5c1ae379" containerName="payload-processing" containerID="cri-o://4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7" gracePeriod=25 Nov 02 15:35:35.625820 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.625745 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:35:35.759696 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.759662 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a08d9f-a36e-4a36-90ef-414f5c1ae379-kube-api-access-cvpsg\" (UniqueName: \"kubernetes.io/projected/06a08d9f-a36e-4a36-90ef-414f5c1ae379-kube-api-access-cvpsg\") pod \"06a08d9f-a36e-4a36-90ef-414f5c1ae379\" (UID: \"06a08d9f-a36e-4a36-90ef-414f5c1ae379\") " Nov 02 15:35:35.759866 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.759827 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a08d9f-a36e-4a36-90ef-414f5c1ae379-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a08d9f-a36e-4a36-90ef-414f5c1ae379-plugins-config-volume\") pod \"06a08d9f-a36e-4a36-90ef-414f5c1ae379\" (UID: \"06a08d9f-a36e-4a36-90ef-414f5c1ae379\") " Nov 02 15:35:35.760226 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.760199 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a08d9f-a36e-4a36-90ef-414f5c1ae379-plugins-config-volume" pod "06a08d9f-a36e-4a36-90ef-414f5c1ae379" (UID: "06a08d9f-a36e-4a36-90ef-414f5c1ae379"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:35:35.760365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.760347 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06a08d9f-a36e-4a36-90ef-414f5c1ae379-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:35.761786 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.761767 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a08d9f-a36e-4a36-90ef-414f5c1ae379-kube-api-access-cvpsg" pod "06a08d9f-a36e-4a36-90ef-414f5c1ae379" (UID: "06a08d9f-a36e-4a36-90ef-414f5c1ae379"). InnerVolumeSpecName "kube-api-access-cvpsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:35:35.861553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.861522 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvpsg\" (UniqueName: \"kubernetes.io/projected/06a08d9f-a36e-4a36-90ef-414f5c1ae379-kube-api-access-cvpsg\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:35:35.933485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.933409 5476 generic.go:356] "Generic (PLEG): container finished" podID="06a08d9f-a36e-4a36-90ef-414f5c1ae379" containerID="4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7" exitCode=0 Nov 02 15:35:35.933485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.933440 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" Nov 02 15:35:35.933665 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.933484 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" event={"ID":"06a08d9f-a36e-4a36-90ef-414f5c1ae379","Type":"ContainerDied","Data":"4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7"} Nov 02 15:35:35.933665 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.933522 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9" event={"ID":"06a08d9f-a36e-4a36-90ef-414f5c1ae379","Type":"ContainerDied","Data":"a8235727da04b1f9336e8012483b4242704ad99f68247a4411f5722a2fb83053"} Nov 02 15:35:35.933665 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.933544 5476 scope.go:122] "RemoveContainer" containerID="4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7" Nov 02 15:35:35.948060 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.948041 5476 scope.go:122] "RemoveContainer" containerID="4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7" Nov 02 15:35:35.948373 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:35:35.948348 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7\": container with ID starting with 4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7 not found: ID does not exist" containerID="4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7" Nov 02 15:35:35.948475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:35.948377 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7"} err="failed to get container status \"4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7\": rpc error: code = NotFound desc = could not find container \"4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7\": container with ID starting with 4878d9a280a192b059a26a499e9880b13c4c266c29c7ab63e1bce0e0330a8cf7 not found: ID does not exist" Nov 02 15:35:36.089831 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:36.089765 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9"] Nov 02 15:35:36.093317 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:36.093292 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-fb80ca-5757b68f87-wdgc9"] Nov 02 15:35:37.923950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:37.923914 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a08d9f-a36e-4a36-90ef-414f5c1ae379" path="/var/lib/kubelet/pods/06a08d9f-a36e-4a36-90ef-414f5c1ae379/volumes" Nov 02 15:35:40.947539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:40.947500 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg"] Nov 02 15:35:40.948177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:40.948162 5476 cpu_manager.go:414] "containerMap: removing container" podUID="06a08d9f-a36e-4a36-90ef-414f5c1ae379" containerName="payload-processing" Nov 02 15:35:40.948177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:40.948178 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a08d9f-a36e-4a36-90ef-414f5c1ae379" containerName="payload-processing" Nov 02 15:35:40.948342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:40.948195 5476 cpu_manager.go:414] "containerMap: removing container" podUID="1963f89a-b30d-4e51-abd7-b4407a986981" containerName="revoke-keys" Nov 02 15:35:40.948342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:40.948201 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1963f89a-b30d-4e51-abd7-b4407a986981" containerName="revoke-keys" Nov 02 15:35:40.948342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:40.948327 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a08d9f-a36e-4a36-90ef-414f5c1ae379" containerName="payload-processing" Nov 02 15:35:40.948342 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:40.948340 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="1963f89a-b30d-4e51-abd7-b4407a986981" containerName="revoke-keys" Nov 02 15:35:40.951760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:40.951743 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.039530 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.039498 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.039708 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.039674 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkwlp\" (UniqueName: \"kubernetes.io/projected/9a518198-bfea-4c7b-9696-fe657ab1cf90-kube-api-access-tkwlp\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.039858 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.039836 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-metrics-certs\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.040066 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.040042 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a518198-bfea-4c7b-9696-fe657ab1cf90-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.049549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.049523 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-aa5832-serving-cert\"" Nov 02 15:35:41.053080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.053061 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg"] Nov 02 15:35:41.141789 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.141753 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-metrics-certs\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.141982 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.141844 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a518198-bfea-4c7b-9696-fe657ab1cf90-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.141982 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.141875 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.141982 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.141953 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkwlp\" (UniqueName: \"kubernetes.io/projected/9a518198-bfea-4c7b-9696-fe657ab1cf90-kube-api-access-tkwlp\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.142311 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:35:41.142088 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-serving-cert: secret "maas-api-e2e-inf-a-aa5832-serving-cert" not found Nov 02 15:35:41.142311 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:35:41.142154 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-maas-api-tls podName:9a518198-bfea-4c7b-9696-fe657ab1cf90 nodeName:}" failed. No retries permitted until 2026-11-02 15:35:41.642133987 +0000 UTC m=+2243.845012042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-maas-api-tls") pod "maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" (UID: "9a518198-bfea-4c7b-9696-fe657ab1cf90") : secret "maas-api-e2e-inf-a-aa5832-serving-cert" not found Nov 02 15:35:41.142593 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.142567 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a518198-bfea-4c7b-9696-fe657ab1cf90-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.144247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.144227 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-metrics-certs\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.199607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.199544 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkwlp\" (UniqueName: \"kubernetes.io/projected/9a518198-bfea-4c7b-9696-fe657ab1cf90-kube-api-access-tkwlp\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.649067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.649037 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.651377 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.651354 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-maas-api-tls\") pod \"maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:41.864769 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:41.864725 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:42.240560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:42.240476 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg"] Nov 02 15:35:42.245486 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:35:42.245457 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a518198_bfea_4c7b_9696_fe657ab1cf90.slice/crio-d899d1cd019531326c6ade114b6000a776971be4295efeaa8023fe6ed800ad82 WatchSource:0}: Error finding container d899d1cd019531326c6ade114b6000a776971be4295efeaa8023fe6ed800ad82: Status 404 returned error can't find the container with id d899d1cd019531326c6ade114b6000a776971be4295efeaa8023fe6ed800ad82 Nov 02 15:35:42.247221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:42.247194 5476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:35:42.986168 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:42.986129 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" event={"ID":"9a518198-bfea-4c7b-9696-fe657ab1cf90","Type":"ContainerStarted","Data":"d899d1cd019531326c6ade114b6000a776971be4295efeaa8023fe6ed800ad82"} Nov 02 15:35:43.994932 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:43.994888 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" event={"ID":"9a518198-bfea-4c7b-9696-fe657ab1cf90","Type":"ContainerStarted","Data":"090418e51076c0aa70b97fd7208730222ffd1667d7936eadb71ac2357be004ab"} Nov 02 15:35:43.995347 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:43.994954 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:44.110178 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:44.110131 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" podStartSLOduration=3.208343852 podStartE2EDuration="4.11011717s" totalImagesPullingTime="901.773318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:35:44.105662949 +0000 UTC m=+2246.308541009" watchObservedRunningTime="2026-11-02 15:35:44.11011717 +0000 UTC m=+2246.312995282" Nov 02 15:35:50.016603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:50.016572 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:35:50.174148 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:35:50.174094 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:35:53.195953 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:53.195909 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj"] Nov 02 15:35:53.213717 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:53.213685 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ca6972-qc7vj"] Nov 02 15:35:53.926144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:35:53.926112 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4ed5ac-b2b6-4880-b219-26863a2d41f4" path="/var/lib/kubelet/pods/1c4ed5ac-b2b6-4880-b219-26863a2d41f4/volumes" Nov 02 15:36:12.693407 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:12.693369 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97"] Nov 02 15:36:17.697130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:17.697054 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" podUID="a42d5004-cd64-4077-a706-8e7920be3392" containerName="payload-processing" containerID="cri-o://76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7" gracePeriod=25 Nov 02 15:36:18.041530 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.041504 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:36:18.169696 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.169659 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a42d5004-cd64-4077-a706-8e7920be3392-kube-api-access-cwfc9\" (UniqueName: \"kubernetes.io/projected/a42d5004-cd64-4077-a706-8e7920be3392-kube-api-access-cwfc9\") pod \"a42d5004-cd64-4077-a706-8e7920be3392\" (UID: \"a42d5004-cd64-4077-a706-8e7920be3392\") " Nov 02 15:36:18.169859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.169740 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a42d5004-cd64-4077-a706-8e7920be3392-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42d5004-cd64-4077-a706-8e7920be3392-plugins-config-volume\") pod \"a42d5004-cd64-4077-a706-8e7920be3392\" (UID: \"a42d5004-cd64-4077-a706-8e7920be3392\") " Nov 02 15:36:18.170156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.170131 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42d5004-cd64-4077-a706-8e7920be3392-plugins-config-volume" pod "a42d5004-cd64-4077-a706-8e7920be3392" (UID: "a42d5004-cd64-4077-a706-8e7920be3392"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:18.170501 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.170482 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a42d5004-cd64-4077-a706-8e7920be3392-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:18.200427 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.200165 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42d5004-cd64-4077-a706-8e7920be3392-kube-api-access-cwfc9" pod "a42d5004-cd64-4077-a706-8e7920be3392" (UID: "a42d5004-cd64-4077-a706-8e7920be3392"). InnerVolumeSpecName "kube-api-access-cwfc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:18.294075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.292983 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwfc9\" (UniqueName: \"kubernetes.io/projected/a42d5004-cd64-4077-a706-8e7920be3392-kube-api-access-cwfc9\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:18.320816 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.320783 5476 generic.go:356] "Generic (PLEG): container finished" podID="a42d5004-cd64-4077-a706-8e7920be3392" containerID="76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7" exitCode=0 Nov 02 15:36:18.321023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.320844 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" Nov 02 15:36:18.321023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.320865 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" event={"ID":"a42d5004-cd64-4077-a706-8e7920be3392","Type":"ContainerDied","Data":"76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7"} Nov 02 15:36:18.321023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.320967 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97" event={"ID":"a42d5004-cd64-4077-a706-8e7920be3392","Type":"ContainerDied","Data":"53216c4b781a7c7d45a87dd37e48df46f89e5a17a6245ca493da986bc5c532d2"} Nov 02 15:36:18.321023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.321010 5476 scope.go:122] "RemoveContainer" containerID="76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7" Nov 02 15:36:18.332450 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.332427 5476 scope.go:122] "RemoveContainer" containerID="76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7" Nov 02 15:36:18.332829 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:36:18.332808 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7\": container with ID starting with 76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7 not found: ID does not exist" containerID="76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7" Nov 02 15:36:18.332915 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.332839 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7"} err="failed to get container status \"76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7\": rpc error: code = NotFound desc = could not find container \"76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7\": container with ID starting with 76eb7f7d66b767cbe9387cba44c91fcef025f51104c3a22e3fd1b5340da93ae7 not found: ID does not exist" Nov 02 15:36:18.502902 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.502866 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97"] Nov 02 15:36:18.506007 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:18.505971 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-adf096-8b54998df-v6w97"] Nov 02 15:36:19.925948 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:19.925918 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a42d5004-cd64-4077-a706-8e7920be3392" path="/var/lib/kubelet/pods/a42d5004-cd64-4077-a706-8e7920be3392/volumes" Nov 02 15:36:20.146409 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:36:20.146376 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:36:27.429868 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:27.429837 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f"] Nov 02 15:36:27.475986 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:27.475957 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-1c449c-n9w9f"] Nov 02 15:36:27.923603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:27.923568 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5525cf5a-9372-47b7-b913-1e172d9235b4" path="/var/lib/kubelet/pods/5525cf5a-9372-47b7-b913-1e172d9235b4/volumes" Nov 02 15:36:39.547607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.547573 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz"] Nov 02 15:36:39.548219 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.548203 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a42d5004-cd64-4077-a706-8e7920be3392" containerName="payload-processing" Nov 02 15:36:39.548282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.548220 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a42d5004-cd64-4077-a706-8e7920be3392" containerName="payload-processing" Nov 02 15:36:39.548336 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.548326 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a42d5004-cd64-4077-a706-8e7920be3392" containerName="payload-processing" Nov 02 15:36:39.553160 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.553142 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.626402 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.626368 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz"] Nov 02 15:36:39.688763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.688728 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.688763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.688770 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.688990 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.688852 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.688990 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.688919 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d61490-e5dd-4591-a3a8-2979db889bd8-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.689097 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.689044 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.689097 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.689074 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w59kg\" (UniqueName: \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-kube-api-access-w59kg\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.689178 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.689094 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.689214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.689192 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.689253 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.689240 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.790493 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.790457 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.790673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.790515 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.790673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.790603 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.790673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.790627 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.790673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.790657 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.790832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.790707 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d61490-e5dd-4591-a3a8-2979db889bd8-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.790832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.790761 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.790832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.790794 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w59kg\" (UniqueName: \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-kube-api-access-w59kg\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.790832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.790820 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.791087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.790910 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.791087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.790922 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.791087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.791058 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.791299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.791193 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.791739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.791715 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d61490-e5dd-4591-a3a8-2979db889bd8-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.792944 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.792927 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.793227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.793210 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.848560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.848486 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.848702 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.848591 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w59kg\" (UniqueName: \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-kube-api-access-w59kg\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:39.867860 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:39.867827 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:40.304227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:40.304191 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz"] Nov 02 15:36:40.308658 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:36:40.308631 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d61490_e5dd_4591_a3a8_2979db889bd8.slice/crio-aa521cb80c311c38de9826a4b98c6b7ec25202c3c8b94ea65f20c007107d18e1 WatchSource:0}: Error finding container aa521cb80c311c38de9826a4b98c6b7ec25202c3c8b94ea65f20c007107d18e1: Status 404 returned error can't find the container with id aa521cb80c311c38de9826a4b98c6b7ec25202c3c8b94ea65f20c007107d18e1 Nov 02 15:36:40.311477 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:40.311445 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:40.311558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:40.311512 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:40.311558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:40.311540 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:40.509813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:40.509770 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" event={"ID":"77d61490-e5dd-4591-a3a8-2979db889bd8","Type":"ContainerStarted","Data":"2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa"} Nov 02 15:36:40.509957 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:40.509838 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" event={"ID":"77d61490-e5dd-4591-a3a8-2979db889bd8","Type":"ContainerStarted","Data":"aa521cb80c311c38de9826a4b98c6b7ec25202c3c8b94ea65f20c007107d18e1"} Nov 02 15:36:40.629538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:40.629480 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" podStartSLOduration=1.629460104 podStartE2EDuration="1.629460104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:40.628688106 +0000 UTC m=+2302.831566165" watchObservedRunningTime="2026-11-02 15:36:40.629460104 +0000 UTC m=+2302.832338163" Nov 02 15:36:40.868057 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:40.867990 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:41.870102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:41.870069 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 15:36:41.870489 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:41.870123 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" podUID="77d61490-e5dd-4591-a3a8-2979db889bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 15:36:42.869050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:42.868979 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 15:36:42.869269 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:42.869076 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" podUID="77d61490-e5dd-4591-a3a8-2979db889bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 15:36:43.869099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:43.869063 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 15:36:43.869607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:43.869119 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" podUID="77d61490-e5dd-4591-a3a8-2979db889bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 15:36:44.869212 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:44.869171 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 15:36:44.869643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:44.869229 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" podUID="77d61490-e5dd-4591-a3a8-2979db889bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 15:36:45.868732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:45.868696 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 02 15:36:45.868921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:45.868751 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" podUID="77d61490-e5dd-4591-a3a8-2979db889bd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 02 15:36:46.872927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:46.872898 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:46.873422 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:46.873400 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:46.874198 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:46.874176 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:36:47.572799 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:47.572766 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z"] Nov 02 15:36:47.671950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:47.671910 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj"] Nov 02 15:36:50.148395 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:36:50.148359 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:36:50.404224 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:50.404137 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v"] Nov 02 15:36:50.449950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:50.449920 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-fb80ca-qt28v"] Nov 02 15:36:51.924290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:51.924257 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1963f89a-b30d-4e51-abd7-b4407a986981" path="/var/lib/kubelet/pods/1963f89a-b30d-4e51-abd7-b4407a986981/volumes" Nov 02 15:36:52.574278 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:52.574234 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" podUID="9b869364-017a-4cdd-8501-c68fcee1d73b" containerName="payload-processing" containerID="cri-o://ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea" gracePeriod=25 Nov 02 15:36:52.673253 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:52.673205 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" podUID="b785048a-948c-4334-a2cb-c12ff8990d24" containerName="payload-pre-processing" containerID="cri-o://147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0" gracePeriod=25 Nov 02 15:36:52.848081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:52.848054 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:36:52.940345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:52.940304 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b869364-017a-4cdd-8501-c68fcee1d73b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b869364-017a-4cdd-8501-c68fcee1d73b-plugins-config-volume\") pod \"9b869364-017a-4cdd-8501-c68fcee1d73b\" (UID: \"9b869364-017a-4cdd-8501-c68fcee1d73b\") " Nov 02 15:36:52.940844 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:52.940456 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b869364-017a-4cdd-8501-c68fcee1d73b-kube-api-access-tx8lb\" (UniqueName: \"kubernetes.io/projected/9b869364-017a-4cdd-8501-c68fcee1d73b-kube-api-access-tx8lb\") pod \"9b869364-017a-4cdd-8501-c68fcee1d73b\" (UID: \"9b869364-017a-4cdd-8501-c68fcee1d73b\") " Nov 02 15:36:52.940926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:52.940836 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b869364-017a-4cdd-8501-c68fcee1d73b-plugins-config-volume" pod "9b869364-017a-4cdd-8501-c68fcee1d73b" (UID: "9b869364-017a-4cdd-8501-c68fcee1d73b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:52.941565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:52.941527 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b869364-017a-4cdd-8501-c68fcee1d73b-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:52.943310 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:52.943283 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b869364-017a-4cdd-8501-c68fcee1d73b-kube-api-access-tx8lb" pod "9b869364-017a-4cdd-8501-c68fcee1d73b" (UID: "9b869364-017a-4cdd-8501-c68fcee1d73b"). InnerVolumeSpecName "kube-api-access-tx8lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:52.964922 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:52.964897 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:36:53.042487 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.042455 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b785048a-948c-4334-a2cb-c12ff8990d24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b785048a-948c-4334-a2cb-c12ff8990d24-plugins-config-volume\") pod \"b785048a-948c-4334-a2cb-c12ff8990d24\" (UID: \"b785048a-948c-4334-a2cb-c12ff8990d24\") " Nov 02 15:36:53.042649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.042534 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b785048a-948c-4334-a2cb-c12ff8990d24-kube-api-access-7c6g4\" (UniqueName: \"kubernetes.io/projected/b785048a-948c-4334-a2cb-c12ff8990d24-kube-api-access-7c6g4\") pod \"b785048a-948c-4334-a2cb-c12ff8990d24\" (UID: \"b785048a-948c-4334-a2cb-c12ff8990d24\") " Nov 02 15:36:53.042829 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.042808 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b785048a-948c-4334-a2cb-c12ff8990d24-plugins-config-volume" pod "b785048a-948c-4334-a2cb-c12ff8990d24" (UID: "b785048a-948c-4334-a2cb-c12ff8990d24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:53.042964 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.042947 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b785048a-948c-4334-a2cb-c12ff8990d24-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:53.043023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.042968 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx8lb\" (UniqueName: \"kubernetes.io/projected/9b869364-017a-4cdd-8501-c68fcee1d73b-kube-api-access-tx8lb\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:53.044519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.044500 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b785048a-948c-4334-a2cb-c12ff8990d24-kube-api-access-7c6g4" pod "b785048a-948c-4334-a2cb-c12ff8990d24" (UID: "b785048a-948c-4334-a2cb-c12ff8990d24"). InnerVolumeSpecName "kube-api-access-7c6g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:53.143890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.143852 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c6g4\" (UniqueName: \"kubernetes.io/projected/b785048a-948c-4334-a2cb-c12ff8990d24-kube-api-access-7c6g4\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:53.612291 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.612257 5476 generic.go:356] "Generic (PLEG): container finished" podID="9b869364-017a-4cdd-8501-c68fcee1d73b" containerID="ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea" exitCode=0 Nov 02 15:36:53.612456 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.612307 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" Nov 02 15:36:53.612456 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.612325 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" event={"ID":"9b869364-017a-4cdd-8501-c68fcee1d73b","Type":"ContainerDied","Data":"ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea"} Nov 02 15:36:53.612456 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.612403 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z" event={"ID":"9b869364-017a-4cdd-8501-c68fcee1d73b","Type":"ContainerDied","Data":"bb41fddf49dd6d37dc4744c6dc836ee1ec337aa34dae29ef41e6994fafb6e108"} Nov 02 15:36:53.612456 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.612433 5476 scope.go:122] "RemoveContainer" containerID="ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea" Nov 02 15:36:53.613956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.613934 5476 generic.go:356] "Generic (PLEG): container finished" podID="b785048a-948c-4334-a2cb-c12ff8990d24" containerID="147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0" exitCode=0 Nov 02 15:36:53.614075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.614034 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" event={"ID":"b785048a-948c-4334-a2cb-c12ff8990d24","Type":"ContainerDied","Data":"147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0"} Nov 02 15:36:53.614075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.614020 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" Nov 02 15:36:53.614204 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.614088 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj" event={"ID":"b785048a-948c-4334-a2cb-c12ff8990d24","Type":"ContainerDied","Data":"b9d262174dd27d380e6bfbbeeebe4e331762bce6eae4f2288d2014e05743a4c7"} Nov 02 15:36:53.625141 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.625120 5476 scope.go:122] "RemoveContainer" containerID="ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea" Nov 02 15:36:53.625408 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:36:53.625388 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea\": container with ID starting with ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea not found: ID does not exist" containerID="ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea" Nov 02 15:36:53.625461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.625416 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea"} err="failed to get container status \"ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea\": rpc error: code = NotFound desc = could not find container \"ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea\": container with ID starting with ccb651293b0c8b7e8bb14c1cd11b6cb33040f7dc0448dc62984e367929bbf9ea not found: ID does not exist" Nov 02 15:36:53.625461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.625433 5476 scope.go:122] "RemoveContainer" containerID="147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0" Nov 02 15:36:53.634778 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.634762 5476 scope.go:122] "RemoveContainer" containerID="147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0" Nov 02 15:36:53.635070 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:36:53.635054 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0\": container with ID starting with 147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0 not found: ID does not exist" containerID="147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0" Nov 02 15:36:53.635128 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.635079 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0"} err="failed to get container status \"147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0\": rpc error: code = NotFound desc = could not find container \"147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0\": container with ID starting with 147e2c6a4c1a30079511b5b7ad19460f24e077c171635bc342e0f0985a607de0 not found: ID does not exist" Nov 02 15:36:53.769674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.769640 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z"] Nov 02 15:36:53.816247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.816210 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-5afebc-5cdc47c7dc-kpx2z"] Nov 02 15:36:53.880214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.880177 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj"] Nov 02 15:36:53.884745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.884714 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5afebc-7b74c6fdb7-r2xnj"] Nov 02 15:36:53.924433 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.924403 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b869364-017a-4cdd-8501-c68fcee1d73b" path="/var/lib/kubelet/pods/9b869364-017a-4cdd-8501-c68fcee1d73b/volumes" Nov 02 15:36:53.924810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:53.924797 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b785048a-948c-4334-a2cb-c12ff8990d24" path="/var/lib/kubelet/pods/b785048a-948c-4334-a2cb-c12ff8990d24/volumes" Nov 02 15:36:54.654605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.654568 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb"] Nov 02 15:36:54.655548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.655527 5476 cpu_manager.go:414] "containerMap: removing container" podUID="9b869364-017a-4cdd-8501-c68fcee1d73b" containerName="payload-processing" Nov 02 15:36:54.655610 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.655550 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b869364-017a-4cdd-8501-c68fcee1d73b" containerName="payload-processing" Nov 02 15:36:54.655647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.655614 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b785048a-948c-4334-a2cb-c12ff8990d24" containerName="payload-pre-processing" Nov 02 15:36:54.655647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.655623 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b785048a-948c-4334-a2cb-c12ff8990d24" containerName="payload-pre-processing" Nov 02 15:36:54.655773 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.655761 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b785048a-948c-4334-a2cb-c12ff8990d24" containerName="payload-pre-processing" Nov 02 15:36:54.655808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.655785 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b869364-017a-4cdd-8501-c68fcee1d73b" containerName="payload-processing" Nov 02 15:36:54.660929 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.660910 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.760767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.760738 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb"] Nov 02 15:36:54.767460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.767395 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.767699 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.767587 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kwtj\" (UniqueName: \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-kube-api-access-4kwtj\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.768755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.768136 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.768755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.768185 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.768755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.768224 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.768755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.768248 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.768755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.768302 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.768755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.768362 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.768755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.768403 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.889348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.887938 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.889348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.888008 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.889348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.888053 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.889348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.888084 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.889348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.888146 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.889348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.888212 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.889348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.888254 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.889348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.888331 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.889348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.888476 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kwtj\" (UniqueName: \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-kube-api-access-4kwtj\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.889348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.889321 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.890042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.889571 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.890042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.889807 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.890162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.890056 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.891163 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.891139 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.913294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.908853 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.913294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.909422 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.948860 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.948442 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kwtj\" (UniqueName: \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-kube-api-access-4kwtj\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.965080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.959532 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:54.974810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:54.974777 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:55.420947 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:55.420910 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb"] Nov 02 15:36:55.425164 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:36:55.425136 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1b9353_6505_4328_93c1_4e5aa6c2566a.slice/crio-034382468bb9e60eb95319514c20285d9ad22e320b4e2c7cff2377fe62983f2e WatchSource:0}: Error finding container 034382468bb9e60eb95319514c20285d9ad22e320b4e2c7cff2377fe62983f2e: Status 404 returned error can't find the container with id 034382468bb9e60eb95319514c20285d9ad22e320b4e2c7cff2377fe62983f2e Nov 02 15:36:55.427741 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:55.427706 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:55.427843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:55.427786 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:55.427843 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:55.427832 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:55.632898 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:55.632858 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" event={"ID":"fc1b9353-6505-4328-93c1-4e5aa6c2566a","Type":"ContainerStarted","Data":"869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b"} Nov 02 15:36:55.633187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:55.632927 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" event={"ID":"fc1b9353-6505-4328-93c1-4e5aa6c2566a","Type":"ContainerStarted","Data":"034382468bb9e60eb95319514c20285d9ad22e320b4e2c7cff2377fe62983f2e"} Nov 02 15:36:55.750889 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:55.750774 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" podStartSLOduration=1.7507603569999999 podStartE2EDuration="1.750760357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:55.747678812 +0000 UTC m=+2317.950556885" watchObservedRunningTime="2026-11-02 15:36:55.750760357 +0000 UTC m=+2317.953638416" Nov 02 15:36:55.975649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:55.975615 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:36:56.992024 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:56.989113 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:36:56.992024 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:56.989192 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 15:36:57.747060 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:57.747027 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z"] Nov 02 15:36:57.747323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:57.747266 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" containerID="cri-o://a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88" gracePeriod=30 Nov 02 15:36:57.989041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:57.988113 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:36:57.989041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:57.988176 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 15:36:59.000031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:58.998104 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:36:59.000031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:58.998182 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 15:36:59.989374 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:59.989336 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:36:59.997043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:36:59.996116 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 15:37:00.991019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:00.979376 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:37:00.991019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:00.979434 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 15:37:01.976213 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:01.976170 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:37:01.976446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:01.976241 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 02 15:37:02.982848 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:02.982815 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:37:02.984341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:02.984322 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:37:02.985040 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:02.984986 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:37:03.054562 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.054533 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:37:03.069773 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.069737 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-socket\") pod \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " Nov 02 15:37:03.069911 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.069848 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-certs\") pod \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " Nov 02 15:37:03.070005 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.069915 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istiod-ca-cert\") pod \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " Nov 02 15:37:03.070005 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.069956 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-podinfo\") pod \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " Nov 02 15:37:03.070132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.069974 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-socket" pod "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" (UID: "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:03.070132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.070018 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-data\") pod \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " Nov 02 15:37:03.070132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.070110 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-kube-api-access-6pk9n\" (UniqueName: \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-kube-api-access-6pk9n\") pod \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " Nov 02 15:37:03.070302 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.070173 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-token\" (UniqueName: \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-token\") pod \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " Nov 02 15:37:03.070302 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.070203 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-envoy\") pod \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " Nov 02 15:37:03.070302 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.070211 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-certs" pod "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" (UID: "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:03.070302 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.070241 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-credential-socket\") pod \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\" (UID: \"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87\") " Nov 02 15:37:03.070302 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.070293 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-data" pod "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" (UID: "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:03.070560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.070313 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istiod-ca-cert" pod "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" (UID: "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:03.070695 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.070669 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-credential-socket" pod "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" (UID: "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:03.071113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.071087 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:03.071234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.071118 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:03.071234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.071134 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:03.071234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.071150 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:03.071234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.071164 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:03.072478 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.072458 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-token" pod "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" (UID: "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:03.072577 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.072542 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-kube-api-access-6pk9n" pod "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" (UID: "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87"). InnerVolumeSpecName "kube-api-access-6pk9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:03.072808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.072781 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-envoy" pod "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" (UID: "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:03.073160 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.073133 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-podinfo" pod "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" (UID: "f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:37:03.083285 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.083260 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:37:03.172681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.172649 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:03.172681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.172674 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pk9n\" (UniqueName: \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-kube-api-access-6pk9n\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:03.172681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.172686 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:03.172943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.172695 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:03.837716 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.837684 5476 generic.go:356] "Generic (PLEG): container finished" podID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerID="a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88" exitCode=0 Nov 02 15:37:03.838182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.838160 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" Nov 02 15:37:03.839263 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.839238 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" event={"ID":"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87","Type":"ContainerDied","Data":"a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88"} Nov 02 15:37:03.839662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.839629 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z" event={"ID":"f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87","Type":"ContainerDied","Data":"aef577be96736ad49555033510a1aef17125f9c82e9fb6dd54e4af5200810cba"} Nov 02 15:37:03.839901 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.839871 5476 scope.go:122] "RemoveContainer" containerID="a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88" Nov 02 15:37:03.861832 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.861807 5476 scope.go:122] "RemoveContainer" containerID="f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878" Nov 02 15:37:03.874515 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.874229 5476 scope.go:122] "RemoveContainer" containerID="a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88" Nov 02 15:37:03.874608 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:37:03.874590 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88\": container with ID starting with a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88 not found: ID does not exist" containerID="a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88" Nov 02 15:37:03.874670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.874622 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88"} err="failed to get container status \"a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88\": rpc error: code = NotFound desc = could not find container \"a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88\": container with ID starting with a0e41cb6cc717d0046f9b1eb20f3f6bcc6a10491e2684c413ec677ff19f8bc88 not found: ID does not exist" Nov 02 15:37:03.874670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.874641 5476 scope.go:122] "RemoveContainer" containerID="f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878" Nov 02 15:37:03.874983 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:37:03.874956 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878\": container with ID starting with f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878 not found: ID does not exist" containerID="f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878" Nov 02 15:37:03.875321 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.874982 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878"} err="failed to get container status \"f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878\": rpc error: code = NotFound desc = could not find container \"f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878\": container with ID starting with f1c189b4dc62b87479122c4347137f9d0a3e08c0090f5d4ad8f9e59714110878 not found: ID does not exist" Nov 02 15:37:03.997564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:03.997526 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z"] Nov 02 15:37:04.043498 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:04.043468 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-fb80ca-openshift-default-7b89cbd6d5-pb52z"] Nov 02 15:37:05.924733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:05.924698 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" path="/var/lib/kubelet/pods/f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87/volumes" Nov 02 15:37:07.085511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.085477 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2"] Nov 02 15:37:07.086189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.086172 5476 cpu_manager.go:414] "containerMap: removing container" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" Nov 02 15:37:07.086189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.086188 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" Nov 02 15:37:07.086324 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.086316 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" Nov 02 15:37:07.086393 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.086339 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" Nov 02 15:37:07.091743 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.091717 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:37:07.144338 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.144306 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2"] Nov 02 15:37:07.186463 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.186425 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2c35ee\"" Nov 02 15:37:07.186669 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.186600 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2c35ee-dockercfg-4zkkx\"" Nov 02 15:37:07.193513 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.192193 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2\" (UID: \"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:37:07.193513 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.192304 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64cj8\" (UniqueName: \"kubernetes.io/projected/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-kube-api-access-64cj8\") pod \"payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2\" (UID: \"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:37:07.302200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.295391 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2\" (UID: \"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:37:07.302200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.295519 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cj8\" (UniqueName: \"kubernetes.io/projected/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-kube-api-access-64cj8\") pod \"payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2\" (UID: \"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:37:07.302200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.296765 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2\" (UID: \"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:37:07.353593 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.353569 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64cj8\" (UniqueName: \"kubernetes.io/projected/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-kube-api-access-64cj8\") pod \"payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2\" (UID: \"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:37:07.407849 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.407815 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:37:07.781559 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.781477 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2"] Nov 02 15:37:07.786263 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:37:07.786229 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d123955_d5a6_4b5c_9f09_62fe07a5ddb5.slice/crio-2c754361110d8823134d7036c98db3bc7ae95f0745fc60070f091bebabb3f809 WatchSource:0}: Error finding container 2c754361110d8823134d7036c98db3bc7ae95f0745fc60070f091bebabb3f809: Status 404 returned error can't find the container with id 2c754361110d8823134d7036c98db3bc7ae95f0745fc60070f091bebabb3f809 Nov 02 15:37:07.877660 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:07.877630 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" event={"ID":"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5","Type":"ContainerStarted","Data":"2c754361110d8823134d7036c98db3bc7ae95f0745fc60070f091bebabb3f809"} Nov 02 15:37:08.887516 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:08.887479 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" event={"ID":"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5","Type":"ContainerStarted","Data":"f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759"} Nov 02 15:37:08.887949 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:08.887547 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:37:08.999029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:08.998930 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" podStartSLOduration=1.998911633 podStartE2EDuration="1.998911633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:08.996826211 +0000 UTC m=+2331.199704272" watchObservedRunningTime="2026-11-02 15:37:08.998911633 +0000 UTC m=+2331.201789693" Nov 02 15:37:18.590477 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:18.590443 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj"] Nov 02 15:37:19.903239 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:19.903212 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:37:20.210825 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:37:20.210739 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:37:23.591835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:23.591790 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" podUID="739b627b-2186-4b9a-bf9c-dd21c6c192b5" containerName="payload-processing" containerID="cri-o://40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7" gracePeriod=25 Nov 02 15:37:23.863390 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:23.863363 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:37:23.949241 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:23.949208 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739b627b-2186-4b9a-bf9c-dd21c6c192b5-kube-api-access-xnrc6\" (UniqueName: \"kubernetes.io/projected/739b627b-2186-4b9a-bf9c-dd21c6c192b5-kube-api-access-xnrc6\") pod \"739b627b-2186-4b9a-bf9c-dd21c6c192b5\" (UID: \"739b627b-2186-4b9a-bf9c-dd21c6c192b5\") " Nov 02 15:37:23.949419 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:23.949333 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739b627b-2186-4b9a-bf9c-dd21c6c192b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739b627b-2186-4b9a-bf9c-dd21c6c192b5-plugins-config-volume\") pod \"739b627b-2186-4b9a-bf9c-dd21c6c192b5\" (UID: \"739b627b-2186-4b9a-bf9c-dd21c6c192b5\") " Nov 02 15:37:23.949708 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:23.949685 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739b627b-2186-4b9a-bf9c-dd21c6c192b5-plugins-config-volume" pod "739b627b-2186-4b9a-bf9c-dd21c6c192b5" (UID: "739b627b-2186-4b9a-bf9c-dd21c6c192b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:23.949838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:23.949822 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739b627b-2186-4b9a-bf9c-dd21c6c192b5-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:23.951331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:23.951307 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739b627b-2186-4b9a-bf9c-dd21c6c192b5-kube-api-access-xnrc6" pod "739b627b-2186-4b9a-bf9c-dd21c6c192b5" (UID: "739b627b-2186-4b9a-bf9c-dd21c6c192b5"). InnerVolumeSpecName "kube-api-access-xnrc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:24.002620 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:24.002582 5476 generic.go:356] "Generic (PLEG): container finished" podID="739b627b-2186-4b9a-bf9c-dd21c6c192b5" containerID="40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7" exitCode=0 Nov 02 15:37:24.002894 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:24.002632 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" Nov 02 15:37:24.003102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:24.002651 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" event={"ID":"739b627b-2186-4b9a-bf9c-dd21c6c192b5","Type":"ContainerDied","Data":"40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7"} Nov 02 15:37:24.003207 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:24.003141 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj" event={"ID":"739b627b-2186-4b9a-bf9c-dd21c6c192b5","Type":"ContainerDied","Data":"0ca3ce9904452f0ae1f2789d45c64b430559154fc24902cddd576a08b6dfa191"} Nov 02 15:37:24.003207 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:24.003181 5476 scope.go:122] "RemoveContainer" containerID="40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7" Nov 02 15:37:24.028613 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:24.028590 5476 scope.go:122] "RemoveContainer" containerID="40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7" Nov 02 15:37:24.029033 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:37:24.029008 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7\": container with ID starting with 40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7 not found: ID does not exist" containerID="40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7" Nov 02 15:37:24.029117 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:24.029044 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7"} err="failed to get container status \"40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7\": rpc error: code = NotFound desc = could not find container \"40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7\": container with ID starting with 40eb519523b9d650addd0695fb0fbf5f5a5116f59bf5444453b982069a7352a7 not found: ID does not exist" Nov 02 15:37:24.051187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:24.051156 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnrc6\" (UniqueName: \"kubernetes.io/projected/739b627b-2186-4b9a-bf9c-dd21c6c192b5-kube-api-access-xnrc6\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:24.159083 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:24.159040 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj"] Nov 02 15:37:24.163614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:24.163584 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aa5832-6bbfc4f6fc-q44tj"] Nov 02 15:37:25.923681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:25.923646 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739b627b-2186-4b9a-bf9c-dd21c6c192b5" path="/var/lib/kubelet/pods/739b627b-2186-4b9a-bf9c-dd21c6c192b5/volumes" Nov 02 15:37:39.263069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:39.262953 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr"] Nov 02 15:37:39.364824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:39.364787 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr"] Nov 02 15:37:44.263482 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.263431 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" podUID="6c4d8faa-f354-4fb8-9091-51ed0f1f29ff" containerName="payload-processing" containerID="cri-o://6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465" gracePeriod=25 Nov 02 15:37:44.365598 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.365546 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" podUID="da89a1aa-673c-4856-b66d-cf095a3d966b" containerName="payload-pre-processing" containerID="cri-o://56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c" gracePeriod=25 Nov 02 15:37:44.530019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.529914 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:37:44.646098 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.646067 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-plugins-config-volume\") pod \"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff\" (UID: \"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff\") " Nov 02 15:37:44.646250 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.646146 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-kube-api-access-xb4w2\" (UniqueName: \"kubernetes.io/projected/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-kube-api-access-xb4w2\") pod \"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff\" (UID: \"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff\") " Nov 02 15:37:44.646526 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.646499 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-plugins-config-volume" pod "6c4d8faa-f354-4fb8-9091-51ed0f1f29ff" (UID: "6c4d8faa-f354-4fb8-9091-51ed0f1f29ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:44.647046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.646848 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:44.648504 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.648475 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-kube-api-access-xb4w2" pod "6c4d8faa-f354-4fb8-9091-51ed0f1f29ff" (UID: "6c4d8faa-f354-4fb8-9091-51ed0f1f29ff"). InnerVolumeSpecName "kube-api-access-xb4w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:44.652934 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.652908 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:37:44.748027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.747966 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da89a1aa-673c-4856-b66d-cf095a3d966b-kube-api-access-c2dzs\" (UniqueName: \"kubernetes.io/projected/da89a1aa-673c-4856-b66d-cf095a3d966b-kube-api-access-c2dzs\") pod \"da89a1aa-673c-4856-b66d-cf095a3d966b\" (UID: \"da89a1aa-673c-4856-b66d-cf095a3d966b\") " Nov 02 15:37:44.748215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.748107 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da89a1aa-673c-4856-b66d-cf095a3d966b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da89a1aa-673c-4856-b66d-cf095a3d966b-plugins-config-volume\") pod \"da89a1aa-673c-4856-b66d-cf095a3d966b\" (UID: \"da89a1aa-673c-4856-b66d-cf095a3d966b\") " Nov 02 15:37:44.748475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.748452 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da89a1aa-673c-4856-b66d-cf095a3d966b-plugins-config-volume" pod "da89a1aa-673c-4856-b66d-cf095a3d966b" (UID: "da89a1aa-673c-4856-b66d-cf095a3d966b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:44.748731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.748716 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da89a1aa-673c-4856-b66d-cf095a3d966b-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:44.748781 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.748737 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb4w2\" (UniqueName: \"kubernetes.io/projected/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff-kube-api-access-xb4w2\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:44.749970 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.749949 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da89a1aa-673c-4856-b66d-cf095a3d966b-kube-api-access-c2dzs" pod "da89a1aa-673c-4856-b66d-cf095a3d966b" (UID: "da89a1aa-673c-4856-b66d-cf095a3d966b"). InnerVolumeSpecName "kube-api-access-c2dzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:44.850102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:44.850023 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2dzs\" (UniqueName: \"kubernetes.io/projected/da89a1aa-673c-4856-b66d-cf095a3d966b-kube-api-access-c2dzs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:45.191130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.191094 5476 generic.go:356] "Generic (PLEG): container finished" podID="6c4d8faa-f354-4fb8-9091-51ed0f1f29ff" containerID="6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465" exitCode=0 Nov 02 15:37:45.191345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.191143 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" Nov 02 15:37:45.191345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.191179 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" event={"ID":"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff","Type":"ContainerDied","Data":"6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465"} Nov 02 15:37:45.191345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.191244 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr" event={"ID":"6c4d8faa-f354-4fb8-9091-51ed0f1f29ff","Type":"ContainerDied","Data":"fec95cb85004f6f39e26a76c062d91c78d76275e872fa99cd4d77365b5d0c2de"} Nov 02 15:37:45.191345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.191274 5476 scope.go:122] "RemoveContainer" containerID="6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465" Nov 02 15:37:45.192767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.192741 5476 generic.go:356] "Generic (PLEG): container finished" podID="da89a1aa-673c-4856-b66d-cf095a3d966b" containerID="56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c" exitCode=0 Nov 02 15:37:45.192888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.192783 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" Nov 02 15:37:45.192888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.192819 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" event={"ID":"da89a1aa-673c-4856-b66d-cf095a3d966b","Type":"ContainerDied","Data":"56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c"} Nov 02 15:37:45.192888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.192866 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr" event={"ID":"da89a1aa-673c-4856-b66d-cf095a3d966b","Type":"ContainerDied","Data":"b2694fee4732cc9ea6ec37d2b3608b5004ba855a25d9175624746f2e5cdb717b"} Nov 02 15:37:45.202949 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.202924 5476 scope.go:122] "RemoveContainer" containerID="6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465" Nov 02 15:37:45.203288 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:37:45.203265 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465\": container with ID starting with 6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465 not found: ID does not exist" containerID="6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465" Nov 02 15:37:45.203367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.203294 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465"} err="failed to get container status \"6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465\": rpc error: code = NotFound desc = could not find container \"6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465\": container with ID starting with 6161f4b456fc0bb4da201a2943444bd06998e14f530a852fbc8b9e22b31de465 not found: ID does not exist" Nov 02 15:37:45.203367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.203311 5476 scope.go:122] "RemoveContainer" containerID="56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c" Nov 02 15:37:45.213578 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.213563 5476 scope.go:122] "RemoveContainer" containerID="56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c" Nov 02 15:37:45.213854 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:37:45.213835 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c\": container with ID starting with 56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c not found: ID does not exist" containerID="56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c" Nov 02 15:37:45.213927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.213861 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c"} err="failed to get container status \"56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c\": rpc error: code = NotFound desc = could not find container \"56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c\": container with ID starting with 56bd626ad4220807a43cb1a1aa923ff22d09b5ef9beb9e7fc059b402b6d78a7c not found: ID does not exist" Nov 02 15:37:45.347102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.347077 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr"] Nov 02 15:37:45.350241 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.350218 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-822f77-7cd4bf658c-mvplr"] Nov 02 15:37:45.451141 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.451072 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr"] Nov 02 15:37:45.455352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.455326 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-822f77-766858fc97-692nr"] Nov 02 15:37:45.925886 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.925852 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c4d8faa-f354-4fb8-9091-51ed0f1f29ff" path="/var/lib/kubelet/pods/6c4d8faa-f354-4fb8-9091-51ed0f1f29ff/volumes" Nov 02 15:37:45.926314 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:45.926300 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da89a1aa-673c-4856-b66d-cf095a3d966b" path="/var/lib/kubelet/pods/da89a1aa-673c-4856-b66d-cf095a3d966b/volumes" Nov 02 15:37:49.804276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.804240 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4"] Nov 02 15:37:49.805290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.805037 5476 cpu_manager.go:414] "containerMap: removing container" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" Nov 02 15:37:49.805290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.805054 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8291c3e-4c32-43ef-a0d3-c4ed85b1eb87" containerName="istio-proxy" Nov 02 15:37:49.805290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.805068 5476 cpu_manager.go:414] "containerMap: removing container" podUID="da89a1aa-673c-4856-b66d-cf095a3d966b" containerName="payload-pre-processing" Nov 02 15:37:49.805290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.805074 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da89a1aa-673c-4856-b66d-cf095a3d966b" containerName="payload-pre-processing" Nov 02 15:37:49.805290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.805088 5476 cpu_manager.go:414] "containerMap: removing container" podUID="739b627b-2186-4b9a-bf9c-dd21c6c192b5" containerName="payload-processing" Nov 02 15:37:49.805290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.805093 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739b627b-2186-4b9a-bf9c-dd21c6c192b5" containerName="payload-processing" Nov 02 15:37:49.805290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.805108 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6c4d8faa-f354-4fb8-9091-51ed0f1f29ff" containerName="payload-processing" Nov 02 15:37:49.805290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.805113 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c4d8faa-f354-4fb8-9091-51ed0f1f29ff" containerName="payload-processing" Nov 02 15:37:49.805290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.805204 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="739b627b-2186-4b9a-bf9c-dd21c6c192b5" containerName="payload-processing" Nov 02 15:37:49.805290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.805214 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c4d8faa-f354-4fb8-9091-51ed0f1f29ff" containerName="payload-processing" Nov 02 15:37:49.805290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.805231 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="da89a1aa-673c-4856-b66d-cf095a3d966b" containerName="payload-pre-processing" Nov 02 15:37:49.809897 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.809878 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:49.905639 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.905600 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4"] Nov 02 15:37:49.908239 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.908215 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9ca2cf-openshift-default-dockercfg-6ld4m\"" Nov 02 15:37:49.927563 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.927526 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-podinfo\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:49.927721 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.927691 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-data\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:49.927774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.927738 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4m6\" (UniqueName: \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-kube-api-access-4f4m6\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:49.927945 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.927924 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-token\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:49.928089 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.927972 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-socket\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:49.928170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.928143 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-certs\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:49.928234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.928218 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istiod-ca-cert\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:49.928297 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.928280 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-credential-socket\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:49.928347 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:49.928329 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-envoy\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.029645 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.029612 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-envoy\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.029839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.029687 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-podinfo\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.029839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.029791 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-data\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.029839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.029826 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4m6\" (UniqueName: \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-kube-api-access-4f4m6\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.030054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.029895 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-token\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.030054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.029920 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-socket\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.030054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.029952 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-certs\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.030214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.030063 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istiod-ca-cert\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.030214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.030106 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-credential-socket\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.031607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.030360 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-data\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.031607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.030473 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-credential-socket\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.031607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.030749 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-socket\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.031607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.031032 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-certs\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.031814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.031638 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istiod-ca-cert\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.034290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.034058 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-envoy\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.034290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.034222 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-podinfo\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.094137 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.094069 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-token\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.095649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.095621 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4m6\" (UniqueName: \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-kube-api-access-4f4m6\") pod \"e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.123444 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.123418 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:50.159759 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:37:50.159715 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:37:50.570385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.570354 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4"] Nov 02 15:37:50.610178 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:37:50.610146 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d3cf79_d7b4_4d45_8d6a_bdc99a46531b.slice/crio-81a70289fa5f37906c560bafa9ee7f9789f3323b66672359da80d8a6787b0961 WatchSource:0}: Error finding container 81a70289fa5f37906c560bafa9ee7f9789f3323b66672359da80d8a6787b0961: Status 404 returned error can't find the container with id 81a70289fa5f37906c560bafa9ee7f9789f3323b66672359da80d8a6787b0961 Nov 02 15:37:50.612782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.612752 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:37:50.612873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.612823 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:37:50.612873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:50.612856 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:37:51.246211 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:51.246176 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" event={"ID":"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b","Type":"ContainerStarted","Data":"726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00"} Nov 02 15:37:51.246663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:51.246232 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" event={"ID":"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b","Type":"ContainerStarted","Data":"81a70289fa5f37906c560bafa9ee7f9789f3323b66672359da80d8a6787b0961"} Nov 02 15:37:51.362491 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:51.362438 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" podStartSLOduration=2.362423714 podStartE2EDuration="2.362423714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:51.359526591 +0000 UTC m=+2373.562404648" watchObservedRunningTime="2026-11-02 15:37:51.362423714 +0000 UTC m=+2373.565301774" Nov 02 15:37:52.123893 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:52.123859 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:52.129791 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:52.129758 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:52.253711 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:52.253674 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:52.254778 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:52.254756 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:37:55.572707 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:55.572663 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp"] Nov 02 15:37:55.573267 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:55.572960 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" podUID="36ab0548-fe2d-4034-8516-f2c70de24a5d" containerName="maas-api" containerID="cri-o://dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c" gracePeriod=30 Nov 02 15:37:55.884468 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:55.884439 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:37:55.919651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:55.919620 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ab0548-fe2d-4034-8516-f2c70de24a5d-kube-api-access-rfr2k\" (UniqueName: \"kubernetes.io/projected/36ab0548-fe2d-4034-8516-f2c70de24a5d-kube-api-access-rfr2k\") pod \"36ab0548-fe2d-4034-8516-f2c70de24a5d\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " Nov 02 15:37:55.919826 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:55.919763 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-maas-api-tls\") pod \"36ab0548-fe2d-4034-8516-f2c70de24a5d\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " Nov 02 15:37:55.919903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:55.919829 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-metrics-certs\") pod \"36ab0548-fe2d-4034-8516-f2c70de24a5d\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " Nov 02 15:37:55.920203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:55.919975 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ab0548-fe2d-4034-8516-f2c70de24a5d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ab0548-fe2d-4034-8516-f2c70de24a5d-openshift-service-ca\") pod \"36ab0548-fe2d-4034-8516-f2c70de24a5d\" (UID: \"36ab0548-fe2d-4034-8516-f2c70de24a5d\") " Nov 02 15:37:55.920502 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:55.920459 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ab0548-fe2d-4034-8516-f2c70de24a5d-openshift-service-ca" pod "36ab0548-fe2d-4034-8516-f2c70de24a5d" (UID: "36ab0548-fe2d-4034-8516-f2c70de24a5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:55.922289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:55.922256 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ab0548-fe2d-4034-8516-f2c70de24a5d-kube-api-access-rfr2k" pod "36ab0548-fe2d-4034-8516-f2c70de24a5d" (UID: "36ab0548-fe2d-4034-8516-f2c70de24a5d"). InnerVolumeSpecName "kube-api-access-rfr2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:55.923061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:55.923035 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-metrics-certs" pod "36ab0548-fe2d-4034-8516-f2c70de24a5d" (UID: "36ab0548-fe2d-4034-8516-f2c70de24a5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:37:55.923462 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:55.923442 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-maas-api-tls" pod "36ab0548-fe2d-4034-8516-f2c70de24a5d" (UID: "36ab0548-fe2d-4034-8516-f2c70de24a5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:37:56.041017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.035603 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:56.041017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.035641 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36ab0548-fe2d-4034-8516-f2c70de24a5d-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:56.041017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.035657 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ab0548-fe2d-4034-8516-f2c70de24a5d-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:56.041017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.035673 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfr2k\" (UniqueName: \"kubernetes.io/projected/36ab0548-fe2d-4034-8516-f2c70de24a5d-kube-api-access-rfr2k\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:56.295251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.295215 5476 generic.go:356] "Generic (PLEG): container finished" podID="36ab0548-fe2d-4034-8516-f2c70de24a5d" containerID="dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c" exitCode=0 Nov 02 15:37:56.295460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.295261 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" Nov 02 15:37:56.295460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.295292 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" event={"ID":"36ab0548-fe2d-4034-8516-f2c70de24a5d","Type":"ContainerDied","Data":"dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c"} Nov 02 15:37:56.295460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.295348 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp" event={"ID":"36ab0548-fe2d-4034-8516-f2c70de24a5d","Type":"ContainerDied","Data":"6b2ed8af5d7aac047f25093819f575432dddc1b0c3644e0c7b074ccce0b81245"} Nov 02 15:37:56.295460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.295372 5476 scope.go:122] "RemoveContainer" containerID="dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c" Nov 02 15:37:56.307627 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.307611 5476 scope.go:122] "RemoveContainer" containerID="dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c" Nov 02 15:37:56.307896 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:37:56.307876 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c\": container with ID starting with dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c not found: ID does not exist" containerID="dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c" Nov 02 15:37:56.307984 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.307904 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c"} err="failed to get container status \"dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c\": rpc error: code = NotFound desc = could not find container \"dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c\": container with ID starting with dbcf226876d484077405f4ec9dd2734a4ed20323254c37c46d8f2d1e1ca5eb9c not found: ID does not exist" Nov 02 15:37:56.450591 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.450556 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp"] Nov 02 15:37:56.456256 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:56.456232 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-5afebc-65bdcdc865-bzddp"] Nov 02 15:37:57.927602 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:57.927567 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ab0548-fe2d-4034-8516-f2c70de24a5d" path="/var/lib/kubelet/pods/36ab0548-fe2d-4034-8516-f2c70de24a5d/volumes" Nov 02 15:37:57.982216 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:57.982179 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq"] Nov 02 15:37:57.983054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:57.983034 5476 cpu_manager.go:414] "containerMap: removing container" podUID="36ab0548-fe2d-4034-8516-f2c70de24a5d" containerName="maas-api" Nov 02 15:37:57.983140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:57.983055 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ab0548-fe2d-4034-8516-f2c70de24a5d" containerName="maas-api" Nov 02 15:37:57.983255 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:57.983242 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ab0548-fe2d-4034-8516-f2c70de24a5d" containerName="maas-api" Nov 02 15:37:57.988675 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:57.988654 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:37:58.041361 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.041334 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq"] Nov 02 15:37:58.087047 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.085331 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9ca2cf-dockercfg-grvd6\"" Nov 02 15:37:58.087047 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.085435 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9ca2cf\"" Nov 02 15:37:58.180590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.180501 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62a8958-cbe4-431c-aaa6-e2644e34007f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq\" (UID: \"f62a8958-cbe4-431c-aaa6-e2644e34007f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:37:58.180927 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.180897 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsmpw\" (UniqueName: \"kubernetes.io/projected/f62a8958-cbe4-431c-aaa6-e2644e34007f-kube-api-access-qsmpw\") pod \"payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq\" (UID: \"f62a8958-cbe4-431c-aaa6-e2644e34007f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:37:58.238021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.231074 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb"] Nov 02 15:37:58.238021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.237087 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:37:58.254066 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.247077 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb"] Nov 02 15:37:58.284367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.284323 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9g6m\" (UniqueName: \"kubernetes.io/projected/e9223358-7b05-4518-a90f-e12bbaf5c473-kube-api-access-v9g6m\") pod \"payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb\" (UID: \"e9223358-7b05-4518-a90f-e12bbaf5c473\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:37:58.284573 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.284515 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9223358-7b05-4518-a90f-e12bbaf5c473-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb\" (UID: \"e9223358-7b05-4518-a90f-e12bbaf5c473\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:37:58.285235 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.284626 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsmpw\" (UniqueName: \"kubernetes.io/projected/f62a8958-cbe4-431c-aaa6-e2644e34007f-kube-api-access-qsmpw\") pod \"payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq\" (UID: \"f62a8958-cbe4-431c-aaa6-e2644e34007f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:37:58.285235 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.284690 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62a8958-cbe4-431c-aaa6-e2644e34007f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq\" (UID: \"f62a8958-cbe4-431c-aaa6-e2644e34007f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:37:58.285655 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.285628 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62a8958-cbe4-431c-aaa6-e2644e34007f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq\" (UID: \"f62a8958-cbe4-431c-aaa6-e2644e34007f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:37:58.344141 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.344111 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsmpw\" (UniqueName: \"kubernetes.io/projected/f62a8958-cbe4-431c-aaa6-e2644e34007f-kube-api-access-qsmpw\") pod \"payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq\" (UID: \"f62a8958-cbe4-431c-aaa6-e2644e34007f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:37:58.386085 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.386052 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9g6m\" (UniqueName: \"kubernetes.io/projected/e9223358-7b05-4518-a90f-e12bbaf5c473-kube-api-access-v9g6m\") pod \"payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb\" (UID: \"e9223358-7b05-4518-a90f-e12bbaf5c473\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:37:58.386245 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.386168 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9223358-7b05-4518-a90f-e12bbaf5c473-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb\" (UID: \"e9223358-7b05-4518-a90f-e12bbaf5c473\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:37:58.387052 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.387027 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9223358-7b05-4518-a90f-e12bbaf5c473-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb\" (UID: \"e9223358-7b05-4518-a90f-e12bbaf5c473\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:37:58.443807 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.443743 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9g6m\" (UniqueName: \"kubernetes.io/projected/e9223358-7b05-4518-a90f-e12bbaf5c473-kube-api-access-v9g6m\") pod \"payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb\" (UID: \"e9223358-7b05-4518-a90f-e12bbaf5c473\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:37:58.571677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.571645 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:37:58.604240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.604206 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:37:58.953859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:58.953822 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb"] Nov 02 15:37:58.958863 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:37:58.958825 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9223358_7b05_4518_a90f_e12bbaf5c473.slice/crio-fb419c9f4b1f4b88a435c1b6d09eec13dbd6c4f5ff5413502ffedd4750452fe1 WatchSource:0}: Error finding container fb419c9f4b1f4b88a435c1b6d09eec13dbd6c4f5ff5413502ffedd4750452fe1: Status 404 returned error can't find the container with id fb419c9f4b1f4b88a435c1b6d09eec13dbd6c4f5ff5413502ffedd4750452fe1 Nov 02 15:37:59.067320 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:59.067130 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq"] Nov 02 15:37:59.072370 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:37:59.072341 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62a8958_cbe4_431c_aaa6_e2644e34007f.slice/crio-dba3bb370cfc68d8d4e32624577d4f028157d333c55b94a3e3129742dc56fa80 WatchSource:0}: Error finding container dba3bb370cfc68d8d4e32624577d4f028157d333c55b94a3e3129742dc56fa80: Status 404 returned error can't find the container with id dba3bb370cfc68d8d4e32624577d4f028157d333c55b94a3e3129742dc56fa80 Nov 02 15:37:59.322750 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:59.322655 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" event={"ID":"f62a8958-cbe4-431c-aaa6-e2644e34007f","Type":"ContainerStarted","Data":"8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d"} Nov 02 15:37:59.322750 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:59.322712 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" event={"ID":"f62a8958-cbe4-431c-aaa6-e2644e34007f","Type":"ContainerStarted","Data":"dba3bb370cfc68d8d4e32624577d4f028157d333c55b94a3e3129742dc56fa80"} Nov 02 15:37:59.322980 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:59.322767 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:37:59.324518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:59.324489 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" event={"ID":"e9223358-7b05-4518-a90f-e12bbaf5c473","Type":"ContainerStarted","Data":"ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746"} Nov 02 15:37:59.324659 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:59.324533 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" event={"ID":"e9223358-7b05-4518-a90f-e12bbaf5c473","Type":"ContainerStarted","Data":"fb419c9f4b1f4b88a435c1b6d09eec13dbd6c4f5ff5413502ffedd4750452fe1"} Nov 02 15:37:59.324659 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:59.324580 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:37:59.438016 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:59.437943 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" podStartSLOduration=2.437928657 podStartE2EDuration="2.437928657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:59.434509447 +0000 UTC m=+2381.637387507" watchObservedRunningTime="2026-11-02 15:37:59.437928657 +0000 UTC m=+2381.640806757" Nov 02 15:37:59.544114 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:37:59.544056 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" podStartSLOduration=1.5440379480000002 podStartE2EDuration="1.544037948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:59.543277195 +0000 UTC m=+2381.746155256" watchObservedRunningTime="2026-11-02 15:37:59.544037948 +0000 UTC m=+2381.746916002" Nov 02 15:38:06.317050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:06.316987 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x"] Nov 02 15:38:06.317602 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:06.317230 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" containerID="cri-o://d9f8f706203cf777dd982dd33b3b81272f15f11b8da7229d16e448f3852383b0" gracePeriod=30 Nov 02 15:38:10.258618 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:10.258575 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:38:10.333910 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:10.333876 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:38:10.334678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:10.334654 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:38:11.470144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.466883 5476 generic.go:356] "Generic (PLEG): container finished" podID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerID="d9f8f706203cf777dd982dd33b3b81272f15f11b8da7229d16e448f3852383b0" exitCode=0 Nov 02 15:38:11.470144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.466969 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" event={"ID":"a2042e87-2978-48d1-a54f-a6c785cc68dd","Type":"ContainerDied","Data":"d9f8f706203cf777dd982dd33b3b81272f15f11b8da7229d16e448f3852383b0"} Nov 02 15:38:11.470144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.467081 5476 scope.go:122] "RemoveContainer" containerID="4bc155a3dd402aab33edb9b7de65971f3b12072da0087f6a331c1a690c434a65" Nov 02 15:38:11.671487 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.671458 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:38:11.853805 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.853774 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-socket\") pod \"a2042e87-2978-48d1-a54f-a6c785cc68dd\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " Nov 02 15:38:11.854029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.853828 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-podinfo\") pod \"a2042e87-2978-48d1-a54f-a6c785cc68dd\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " Nov 02 15:38:11.854029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.853866 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-data\") pod \"a2042e87-2978-48d1-a54f-a6c785cc68dd\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " Nov 02 15:38:11.854029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.853888 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-envoy\") pod \"a2042e87-2978-48d1-a54f-a6c785cc68dd\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " Nov 02 15:38:11.854029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.853982 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-kube-api-access-j4jcf\" (UniqueName: \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-kube-api-access-j4jcf\") pod \"a2042e87-2978-48d1-a54f-a6c785cc68dd\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " Nov 02 15:38:11.854291 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.854046 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-credential-socket\") pod \"a2042e87-2978-48d1-a54f-a6c785cc68dd\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " Nov 02 15:38:11.854291 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.854080 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-certs\") pod \"a2042e87-2978-48d1-a54f-a6c785cc68dd\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " Nov 02 15:38:11.854291 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.854096 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-token\" (UniqueName: \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-token\") pod \"a2042e87-2978-48d1-a54f-a6c785cc68dd\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " Nov 02 15:38:11.854291 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.854141 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2042e87-2978-48d1-a54f-a6c785cc68dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2042e87-2978-48d1-a54f-a6c785cc68dd-istiod-ca-cert\") pod \"a2042e87-2978-48d1-a54f-a6c785cc68dd\" (UID: \"a2042e87-2978-48d1-a54f-a6c785cc68dd\") " Nov 02 15:38:11.854916 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.854883 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2042e87-2978-48d1-a54f-a6c785cc68dd-istiod-ca-cert" pod "a2042e87-2978-48d1-a54f-a6c785cc68dd" (UID: "a2042e87-2978-48d1-a54f-a6c785cc68dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:11.855312 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.855289 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-data" pod "a2042e87-2978-48d1-a54f-a6c785cc68dd" (UID: "a2042e87-2978-48d1-a54f-a6c785cc68dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.855431 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.855309 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-credential-socket" pod "a2042e87-2978-48d1-a54f-a6c785cc68dd" (UID: "a2042e87-2978-48d1-a54f-a6c785cc68dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.855590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.855528 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-socket" pod "a2042e87-2978-48d1-a54f-a6c785cc68dd" (UID: "a2042e87-2978-48d1-a54f-a6c785cc68dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.855842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.855748 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-certs" pod "a2042e87-2978-48d1-a54f-a6c785cc68dd" (UID: "a2042e87-2978-48d1-a54f-a6c785cc68dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.857452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.857382 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-envoy" pod "a2042e87-2978-48d1-a54f-a6c785cc68dd" (UID: "a2042e87-2978-48d1-a54f-a6c785cc68dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.857553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.857506 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-kube-api-access-j4jcf" pod "a2042e87-2978-48d1-a54f-a6c785cc68dd" (UID: "a2042e87-2978-48d1-a54f-a6c785cc68dd"). InnerVolumeSpecName "kube-api-access-j4jcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:11.857923 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.857898 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-token" pod "a2042e87-2978-48d1-a54f-a6c785cc68dd" (UID: "a2042e87-2978-48d1-a54f-a6c785cc68dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:11.858372 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.858352 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-podinfo" pod "a2042e87-2978-48d1-a54f-a6c785cc68dd" (UID: "a2042e87-2978-48d1-a54f-a6c785cc68dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:38:11.956230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.956198 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.956230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.956223 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.956230 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.956233 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4jcf\" (UniqueName: \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-kube-api-access-j4jcf\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.956465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.956244 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.956465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.956253 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.956465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.956261 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.956465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.956269 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2042e87-2978-48d1-a54f-a6c785cc68dd-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.956465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.956277 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2042e87-2978-48d1-a54f-a6c785cc68dd-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.956465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:11.956285 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2042e87-2978-48d1-a54f-a6c785cc68dd-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:12.500471 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:12.500445 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" Nov 02 15:38:12.501020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:12.500449 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x" event={"ID":"a2042e87-2978-48d1-a54f-a6c785cc68dd","Type":"ContainerDied","Data":"ee3e18c2bd344f42eacca2649552cd80b57a078065a88e1f4e3b1dbc1a23eafa"} Nov 02 15:38:12.501020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:12.500570 5476 scope.go:122] "RemoveContainer" containerID="d9f8f706203cf777dd982dd33b3b81272f15f11b8da7229d16e448f3852383b0" Nov 02 15:38:12.710535 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:12.710494 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x"] Nov 02 15:38:12.714958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:12.714919 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-ck66x"] Nov 02 15:38:13.946778 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:13.946737 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" path="/var/lib/kubelet/pods/a2042e87-2978-48d1-a54f-a6c785cc68dd/volumes" Nov 02 15:38:18.555653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:18.555614 5476 generic.go:356] "Generic (PLEG): container finished" podID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerID="fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8" exitCode=0 Nov 02 15:38:18.555653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:18.555652 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" event={"ID":"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5","Type":"ContainerDied","Data":"fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8"} Nov 02 15:38:18.556262 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:18.556210 5476 scope.go:122] "RemoveContainer" containerID="fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8" Nov 02 15:38:18.558146 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:18.558112 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:38:18.558277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:18.558200 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:38:18.558277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:18.558250 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:38:18.563354 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:18.563331 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:38:18.564736 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:18.564710 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:38:19.565826 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:19.565784 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" event={"ID":"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5","Type":"ContainerStarted","Data":"7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0"} Nov 02 15:38:19.693226 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:19.693101 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:38:19.693432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:19.693353 5476 patch_prober.go:29] interesting pod/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 15:38:19.693432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:19.693412 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 15:38:20.147155 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:38:20.147116 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:38:20.694417 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:20.694378 5476 patch_prober.go:29] interesting pod/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 15:38:20.696163 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:20.696119 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 15:38:21.351721 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.351690 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg"] Nov 02 15:38:21.352413 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.352381 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" Nov 02 15:38:21.352413 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.352407 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" Nov 02 15:38:21.352605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.352444 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" Nov 02 15:38:21.352605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.352450 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" Nov 02 15:38:21.352605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.352578 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" Nov 02 15:38:21.352605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.352599 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2042e87-2978-48d1-a54f-a6c785cc68dd" containerName="istio-proxy" Nov 02 15:38:21.357232 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.357214 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.408666 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.408640 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2c35ee-serving-cert\"" Nov 02 15:38:21.438583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.438553 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5r92\" (UniqueName: \"kubernetes.io/projected/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-kube-api-access-h5r92\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.438761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.438608 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-metrics-certs\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.438857 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.438830 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-maas-api-tls\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.439028 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.438972 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.458442 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.458412 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg"] Nov 02 15:38:21.540511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.540479 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-maas-api-tls\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.540697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.540547 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.540697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.540610 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5r92\" (UniqueName: \"kubernetes.io/projected/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-kube-api-access-h5r92\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.540697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.540643 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-metrics-certs\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.540697 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:38:21.540651 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-serving-cert: secret "maas-api-e2e-api-a-2c35ee-serving-cert" not found Nov 02 15:38:21.540905 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:38:21.540722 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-maas-api-tls podName:679ebea9-1ba7-4eb1-a9f2-784b99f805f4 nodeName:}" failed. No retries permitted until 2026-11-02 15:38:22.040701739 +0000 UTC m=+2404.243579780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-maas-api-tls") pod "maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" (UID: "679ebea9-1ba7-4eb1-a9f2-784b99f805f4") : secret "maas-api-e2e-api-a-2c35ee-serving-cert" not found Nov 02 15:38:21.541313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.541289 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.543261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.543237 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-metrics-certs\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.611074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.603577 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5r92\" (UniqueName: \"kubernetes.io/projected/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-kube-api-access-h5r92\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:21.696411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.696375 5476 patch_prober.go:29] interesting pod/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 15:38:21.698365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:21.696947 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 15:38:22.049080 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:22.048837 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-maas-api-tls\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:22.080736 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:22.080672 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-maas-api-tls\") pod \"maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:22.270601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:22.270401 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:22.697162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:22.696880 5476 patch_prober.go:29] interesting pod/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 15:38:22.697162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:22.696947 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 15:38:22.701041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:22.700777 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg"] Nov 02 15:38:22.701751 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:38:22.701723 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679ebea9_1ba7_4eb1_a9f2_784b99f805f4.slice/crio-4777483ec2f1b7c9849941f3310e4ce445a5b1363614a9c5dac8d476b66596da WatchSource:0}: Error finding container 4777483ec2f1b7c9849941f3310e4ce445a5b1363614a9c5dac8d476b66596da: Status 404 returned error can't find the container with id 4777483ec2f1b7c9849941f3310e4ce445a5b1363614a9c5dac8d476b66596da Nov 02 15:38:23.624054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:23.624020 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" event={"ID":"679ebea9-1ba7-4eb1-a9f2-784b99f805f4","Type":"ContainerStarted","Data":"4777483ec2f1b7c9849941f3310e4ce445a5b1363614a9c5dac8d476b66596da"} Nov 02 15:38:23.699241 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:23.699130 5476 patch_prober.go:29] interesting pod/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 15:38:23.699241 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:23.699189 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 15:38:24.632816 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:24.632777 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" event={"ID":"679ebea9-1ba7-4eb1-a9f2-784b99f805f4","Type":"ContainerStarted","Data":"90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be"} Nov 02 15:38:24.633022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:24.632865 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:24.693118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:24.693081 5476 patch_prober.go:29] interesting pod/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Nov 02 15:38:24.693381 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:24.693352 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Nov 02 15:38:24.753283 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:24.753224 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" podStartSLOduration=2.802187055 podStartE2EDuration="3.753210031s" totalImagesPullingTime="951.022976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:24.752102722 +0000 UTC m=+2406.954980784" watchObservedRunningTime="2026-11-02 15:38:24.753210031 +0000 UTC m=+2406.956088090" Nov 02 15:38:25.697226 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:25.697198 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:38:25.697680 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:25.697657 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:38:25.698492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:25.698476 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:38:30.646131 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:30.646099 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:38:32.421660 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.421628 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg"] Nov 02 15:38:32.422102 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.421841 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" podUID="9a518198-bfea-4c7b-9696-fe657ab1cf90" containerName="maas-api" containerID="cri-o://090418e51076c0aa70b97fd7208730222ffd1667d7936eadb71ac2357be004ab" gracePeriod=30 Nov 02 15:38:32.716669 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.716583 5476 generic.go:356] "Generic (PLEG): container finished" podID="9a518198-bfea-4c7b-9696-fe657ab1cf90" containerID="090418e51076c0aa70b97fd7208730222ffd1667d7936eadb71ac2357be004ab" exitCode=0 Nov 02 15:38:32.716823 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.716726 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" event={"ID":"9a518198-bfea-4c7b-9696-fe657ab1cf90","Type":"ContainerDied","Data":"090418e51076c0aa70b97fd7208730222ffd1667d7936eadb71ac2357be004ab"} Nov 02 15:38:32.749550 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.749523 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:38:32.839784 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.838389 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a518198-bfea-4c7b-9696-fe657ab1cf90-kube-api-access-tkwlp\" (UniqueName: \"kubernetes.io/projected/9a518198-bfea-4c7b-9696-fe657ab1cf90-kube-api-access-tkwlp\") pod \"9a518198-bfea-4c7b-9696-fe657ab1cf90\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " Nov 02 15:38:32.839784 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.838486 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a518198-bfea-4c7b-9696-fe657ab1cf90-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a518198-bfea-4c7b-9696-fe657ab1cf90-openshift-service-ca\") pod \"9a518198-bfea-4c7b-9696-fe657ab1cf90\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " Nov 02 15:38:32.839784 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.838669 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-maas-api-tls\") pod \"9a518198-bfea-4c7b-9696-fe657ab1cf90\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " Nov 02 15:38:32.839784 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.838701 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-metrics-certs\") pod \"9a518198-bfea-4c7b-9696-fe657ab1cf90\" (UID: \"9a518198-bfea-4c7b-9696-fe657ab1cf90\") " Nov 02 15:38:32.856046 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.852768 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a518198-bfea-4c7b-9696-fe657ab1cf90-openshift-service-ca" pod "9a518198-bfea-4c7b-9696-fe657ab1cf90" (UID: "9a518198-bfea-4c7b-9696-fe657ab1cf90"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:32.913015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.910485 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a518198-bfea-4c7b-9696-fe657ab1cf90-kube-api-access-tkwlp" pod "9a518198-bfea-4c7b-9696-fe657ab1cf90" (UID: "9a518198-bfea-4c7b-9696-fe657ab1cf90"). InnerVolumeSpecName "kube-api-access-tkwlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:32.913015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.910940 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-metrics-certs" pod "9a518198-bfea-4c7b-9696-fe657ab1cf90" (UID: "9a518198-bfea-4c7b-9696-fe657ab1cf90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:38:32.917868 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.913468 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-maas-api-tls" pod "9a518198-bfea-4c7b-9696-fe657ab1cf90" (UID: "9a518198-bfea-4c7b-9696-fe657ab1cf90"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:38:32.941194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.941160 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:32.941194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.941194 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a518198-bfea-4c7b-9696-fe657ab1cf90-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:32.941529 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.941216 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkwlp\" (UniqueName: \"kubernetes.io/projected/9a518198-bfea-4c7b-9696-fe657ab1cf90-kube-api-access-tkwlp\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:32.941529 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:32.941233 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a518198-bfea-4c7b-9696-fe657ab1cf90-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:33.727817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:33.727740 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" Nov 02 15:38:33.728278 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:33.727744 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg" event={"ID":"9a518198-bfea-4c7b-9696-fe657ab1cf90","Type":"ContainerDied","Data":"d899d1cd019531326c6ade114b6000a776971be4295efeaa8023fe6ed800ad82"} Nov 02 15:38:33.728278 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:33.727884 5476 scope.go:122] "RemoveContainer" containerID="090418e51076c0aa70b97fd7208730222ffd1667d7936eadb71ac2357be004ab" Nov 02 15:38:33.886194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:33.886165 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg"] Nov 02 15:38:33.932673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:33.932640 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aa5832-56bfbc8754-xvlvg"] Nov 02 15:38:35.932891 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:35.932857 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a518198-bfea-4c7b-9696-fe657ab1cf90" path="/var/lib/kubelet/pods/9a518198-bfea-4c7b-9696-fe657ab1cf90/volumes" Nov 02 15:38:50.161524 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:38:50.161483 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:38:50.270259 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:50.270228 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:38:55.725787 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:55.725752 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf"] Nov 02 15:38:55.830086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:55.830045 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7"] Nov 02 15:38:55.968071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:55.968032 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8"] Nov 02 15:38:55.981154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:38:55.981067 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf"] Nov 02 15:39:00.330514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:00.330478 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz"] Nov 02 15:39:00.330949 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:00.330761 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" podUID="77d61490-e5dd-4591-a3a8-2979db889bd8" containerName="istio-proxy" containerID="cri-o://2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa" gracePeriod=30 Nov 02 15:39:00.357595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:00.357559 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb"] Nov 02 15:39:00.357779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:00.357760 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerName="istio-proxy" containerID="cri-o://869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b" gracePeriod=30 Nov 02 15:39:00.370943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:00.370879 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss"] Nov 02 15:39:00.371197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:00.371136 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" containerID="cri-o://052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3" gracePeriod=30 Nov 02 15:39:00.726622 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:00.726563 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" podUID="7f947d46-4501-4a17-ae94-72793390fd4f" containerName="payload-processing" containerID="cri-o://47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632" gracePeriod=25 Nov 02 15:39:00.830404 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:00.830361 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" podUID="9f01d442-c187-4005-88e0-e36e9cbabc5b" containerName="payload-pre-processing" containerID="cri-o://f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9" gracePeriod=25 Nov 02 15:39:01.876253 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:01.876211 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:39:01.876743 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:01.876284 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" podUID="77d61490-e5dd-4591-a3a8-2979db889bd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:39:02.987090 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:02.987056 5476 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:39:02.987481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:02.987117 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:39:05.689521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.689421 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:39:05.723231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.723185 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:39:05.739033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.739013 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:39:05.740243 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740220 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-kube-api-access-4kwtj\" (UniqueName: \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-kube-api-access-4kwtj\") pod \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " Nov 02 15:39:05.740302 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740282 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-socket\") pod \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " Nov 02 15:39:05.740345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740317 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-credential-socket\") pod \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " Nov 02 15:39:05.740390 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740374 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-envoy\") pod \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " Nov 02 15:39:05.740432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740407 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-data\") pod \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " Nov 02 15:39:05.740487 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740464 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-socket" pod "fc1b9353-6505-4328-93c1-4e5aa6c2566a" (UID: "fc1b9353-6505-4328-93c1-4e5aa6c2566a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.740528 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740488 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-certs\") pod \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " Nov 02 15:39:05.740653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740638 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-token\" (UniqueName: \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-token\") pod \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " Nov 02 15:39:05.740732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740633 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-credential-socket" pod "fc1b9353-6505-4328-93c1-4e5aa6c2566a" (UID: "fc1b9353-6505-4328-93c1-4e5aa6c2566a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.740732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740678 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istiod-ca-cert\") pod \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " Nov 02 15:39:05.740732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740685 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-data" pod "fc1b9353-6505-4328-93c1-4e5aa6c2566a" (UID: "fc1b9353-6505-4328-93c1-4e5aa6c2566a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.740732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740721 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-podinfo\") pod \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\" (UID: \"fc1b9353-6505-4328-93c1-4e5aa6c2566a\") " Nov 02 15:39:05.740938 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.740760 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-certs" pod "fc1b9353-6505-4328-93c1-4e5aa6c2566a" (UID: "fc1b9353-6505-4328-93c1-4e5aa6c2566a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.741054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.741024 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istiod-ca-cert" pod "fc1b9353-6505-4328-93c1-4e5aa6c2566a" (UID: "fc1b9353-6505-4328-93c1-4e5aa6c2566a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:05.741449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.741428 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.741540 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.741455 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.741540 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.741470 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.741540 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.741485 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.741540 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.741500 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.742484 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.742459 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-envoy" pod "fc1b9353-6505-4328-93c1-4e5aa6c2566a" (UID: "fc1b9353-6505-4328-93c1-4e5aa6c2566a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.742652 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.742621 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-token" pod "fc1b9353-6505-4328-93c1-4e5aa6c2566a" (UID: "fc1b9353-6505-4328-93c1-4e5aa6c2566a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.742735 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.742669 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-kube-api-access-4kwtj" pod "fc1b9353-6505-4328-93c1-4e5aa6c2566a" (UID: "fc1b9353-6505-4328-93c1-4e5aa6c2566a"). InnerVolumeSpecName "kube-api-access-4kwtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.742735 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.742679 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-podinfo" pod "fc1b9353-6505-4328-93c1-4e5aa6c2566a" (UID: "fc1b9353-6505-4328-93c1-4e5aa6c2566a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:05.842437 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842407 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-data\") pod \"77d61490-e5dd-4591-a3a8-2979db889bd8\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " Nov 02 15:39:05.842619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842447 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-credential-socket\") pod \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " Nov 02 15:39:05.842619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842479 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-socket\") pod \"77d61490-e5dd-4591-a3a8-2979db889bd8\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " Nov 02 15:39:05.842619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842521 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d61490-e5dd-4591-a3a8-2979db889bd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d61490-e5dd-4591-a3a8-2979db889bd8-istiod-ca-cert\") pod \"77d61490-e5dd-4591-a3a8-2979db889bd8\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " Nov 02 15:39:05.842802 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842632 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-podinfo\") pod \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " Nov 02 15:39:05.842802 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842735 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-data" pod "77d61490-e5dd-4591-a3a8-2979db889bd8" (UID: "77d61490-e5dd-4591-a3a8-2979db889bd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.842802 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842742 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-credential-socket" pod "c3e6b2ff-a84f-428d-9df4-cb020df76d2f" (UID: "c3e6b2ff-a84f-428d-9df4-cb020df76d2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.842802 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842785 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-envoy\") pod \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " Nov 02 15:39:05.842802 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842788 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-socket" pod "77d61490-e5dd-4591-a3a8-2979db889bd8" (UID: "77d61490-e5dd-4591-a3a8-2979db889bd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.843099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842818 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-envoy\") pod \"77d61490-e5dd-4591-a3a8-2979db889bd8\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " Nov 02 15:39:05.843099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842939 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-podinfo\") pod \"77d61490-e5dd-4591-a3a8-2979db889bd8\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " Nov 02 15:39:05.843099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.842989 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d61490-e5dd-4591-a3a8-2979db889bd8-istiod-ca-cert" pod "77d61490-e5dd-4591-a3a8-2979db889bd8" (UID: "77d61490-e5dd-4591-a3a8-2979db889bd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:05.843099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843015 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-socket\") pod \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " Nov 02 15:39:05.843099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843059 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-certs\") pod \"77d61490-e5dd-4591-a3a8-2979db889bd8\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " Nov 02 15:39:05.843364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843114 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-token\" (UniqueName: \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-token\") pod \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " Nov 02 15:39:05.843364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843180 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-kube-api-access-w59kg\" (UniqueName: \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-kube-api-access-w59kg\") pod \"77d61490-e5dd-4591-a3a8-2979db889bd8\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " Nov 02 15:39:05.843364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843219 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-credential-socket\") pod \"77d61490-e5dd-4591-a3a8-2979db889bd8\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " Nov 02 15:39:05.843364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843228 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-socket" pod "c3e6b2ff-a84f-428d-9df4-cb020df76d2f" (UID: "c3e6b2ff-a84f-428d-9df4-cb020df76d2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.843364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843249 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-kube-api-access-l4rds\" (UniqueName: \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-kube-api-access-l4rds\") pod \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " Nov 02 15:39:05.843364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843280 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istiod-ca-cert\") pod \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " Nov 02 15:39:05.843364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843312 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-certs\") pod \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " Nov 02 15:39:05.843730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843383 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-token\" (UniqueName: \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-token\") pod \"77d61490-e5dd-4591-a3a8-2979db889bd8\" (UID: \"77d61490-e5dd-4591-a3a8-2979db889bd8\") " Nov 02 15:39:05.843730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843419 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-data\") pod \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\" (UID: \"c3e6b2ff-a84f-428d-9df4-cb020df76d2f\") " Nov 02 15:39:05.843730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843421 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-certs" pod "77d61490-e5dd-4591-a3a8-2979db889bd8" (UID: "77d61490-e5dd-4591-a3a8-2979db889bd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.843907 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843788 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istiod-ca-cert" pod "c3e6b2ff-a84f-428d-9df4-cb020df76d2f" (UID: "c3e6b2ff-a84f-428d-9df4-cb020df76d2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:05.844037 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.843978 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-data" pod "c3e6b2ff-a84f-428d-9df4-cb020df76d2f" (UID: "c3e6b2ff-a84f-428d-9df4-cb020df76d2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.844037 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844022 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-certs" pod "c3e6b2ff-a84f-428d-9df4-cb020df76d2f" (UID: "c3e6b2ff-a84f-428d-9df4-cb020df76d2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.844215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844052 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-credential-socket" pod "77d61490-e5dd-4591-a3a8-2979db889bd8" (UID: "77d61490-e5dd-4591-a3a8-2979db889bd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.844299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844231 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844255 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844270 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844296 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844312 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844328 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844341 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844354 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844368 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844382 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844396 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844410 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d61490-e5dd-4591-a3a8-2979db889bd8-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844424 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kwtj\" (UniqueName: \"kubernetes.io/projected/fc1b9353-6505-4328-93c1-4e5aa6c2566a-kube-api-access-4kwtj\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.844579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.844440 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc1b9353-6505-4328-93c1-4e5aa6c2566a-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.845453 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.845432 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-envoy" pod "c3e6b2ff-a84f-428d-9df4-cb020df76d2f" (UID: "c3e6b2ff-a84f-428d-9df4-cb020df76d2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.845572 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.845557 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-podinfo" pod "77d61490-e5dd-4591-a3a8-2979db889bd8" (UID: "77d61490-e5dd-4591-a3a8-2979db889bd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:05.845750 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.845730 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-envoy" pod "77d61490-e5dd-4591-a3a8-2979db889bd8" (UID: "77d61490-e5dd-4591-a3a8-2979db889bd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.846236 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.846214 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-podinfo" pod "c3e6b2ff-a84f-428d-9df4-cb020df76d2f" (UID: "c3e6b2ff-a84f-428d-9df4-cb020df76d2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:05.846442 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.846424 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-token" pod "c3e6b2ff-a84f-428d-9df4-cb020df76d2f" (UID: "c3e6b2ff-a84f-428d-9df4-cb020df76d2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.846523 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.846456 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-kube-api-access-w59kg" pod "77d61490-e5dd-4591-a3a8-2979db889bd8" (UID: "77d61490-e5dd-4591-a3a8-2979db889bd8"). InnerVolumeSpecName "kube-api-access-w59kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.847536 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.847516 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-token" pod "77d61490-e5dd-4591-a3a8-2979db889bd8" (UID: "77d61490-e5dd-4591-a3a8-2979db889bd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.847603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.847562 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-kube-api-access-l4rds" pod "c3e6b2ff-a84f-428d-9df4-cb020df76d2f" (UID: "c3e6b2ff-a84f-428d-9df4-cb020df76d2f"). InnerVolumeSpecName "kube-api-access-l4rds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.945911 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.945882 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.946029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.945916 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w59kg\" (UniqueName: \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-kube-api-access-w59kg\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.946029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.945933 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4rds\" (UniqueName: \"kubernetes.io/projected/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-kube-api-access-l4rds\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.946029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.945948 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.946029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.945961 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.946029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.945974 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e6b2ff-a84f-428d-9df4-cb020df76d2f-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.946029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.945988 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.946453 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:05.946434 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d61490-e5dd-4591-a3a8-2979db889bd8-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:06.003738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.003704 5476 generic.go:356] "Generic (PLEG): container finished" podID="77d61490-e5dd-4591-a3a8-2979db889bd8" containerID="2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa" exitCode=0 Nov 02 15:39:06.003943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.003756 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" Nov 02 15:39:06.003943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.003781 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" event={"ID":"77d61490-e5dd-4591-a3a8-2979db889bd8","Type":"ContainerDied","Data":"2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa"} Nov 02 15:39:06.003943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.003836 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz" event={"ID":"77d61490-e5dd-4591-a3a8-2979db889bd8","Type":"ContainerDied","Data":"aa521cb80c311c38de9826a4b98c6b7ec25202c3c8b94ea65f20c007107d18e1"} Nov 02 15:39:06.003943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.003868 5476 scope.go:122] "RemoveContainer" containerID="2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa" Nov 02 15:39:06.005836 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.005810 5476 generic.go:356] "Generic (PLEG): container finished" podID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerID="869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b" exitCode=0 Nov 02 15:39:06.006029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.005926 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" event={"ID":"fc1b9353-6505-4328-93c1-4e5aa6c2566a","Type":"ContainerDied","Data":"869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b"} Nov 02 15:39:06.006029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.005975 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" event={"ID":"fc1b9353-6505-4328-93c1-4e5aa6c2566a","Type":"ContainerDied","Data":"034382468bb9e60eb95319514c20285d9ad22e320b4e2c7cff2377fe62983f2e"} Nov 02 15:39:06.006164 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.006029 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb" Nov 02 15:39:06.009827 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.009238 5476 generic.go:356] "Generic (PLEG): container finished" podID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerID="052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3" exitCode=0 Nov 02 15:39:06.009827 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.009294 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" event={"ID":"c3e6b2ff-a84f-428d-9df4-cb020df76d2f","Type":"ContainerDied","Data":"052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3"} Nov 02 15:39:06.009827 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.009329 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" Nov 02 15:39:06.009827 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.009332 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss" event={"ID":"c3e6b2ff-a84f-428d-9df4-cb020df76d2f","Type":"ContainerDied","Data":"016c58fcec885be7b4950309f9d0ce3ce702bffddf65872d8c91322c908a141d"} Nov 02 15:39:06.017777 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.017759 5476 scope.go:122] "RemoveContainer" containerID="2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa" Nov 02 15:39:06.018081 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:06.018060 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa\": container with ID starting with 2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa not found: ID does not exist" containerID="2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa" Nov 02 15:39:06.018147 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.018088 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa"} err="failed to get container status \"2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa\": rpc error: code = NotFound desc = could not find container \"2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa\": container with ID starting with 2a0791b8ec78adfe6059801006e89d232aedc99fbb624dd2a52373e3c8f9d9fa not found: ID does not exist" Nov 02 15:39:06.018147 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.018104 5476 scope.go:122] "RemoveContainer" containerID="869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b" Nov 02 15:39:06.027632 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.027616 5476 scope.go:122] "RemoveContainer" containerID="869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b" Nov 02 15:39:06.027932 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:06.027910 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b\": container with ID starting with 869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b not found: ID does not exist" containerID="869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b" Nov 02 15:39:06.028067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.027937 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b"} err="failed to get container status \"869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b\": rpc error: code = NotFound desc = could not find container \"869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b\": container with ID starting with 869aa28b5b8d993e8b69f5ae9b282f6f9cc980294c5a51bdcacdfec91835724b not found: ID does not exist" Nov 02 15:39:06.028067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.027956 5476 scope.go:122] "RemoveContainer" containerID="052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3" Nov 02 15:39:06.039973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.039946 5476 scope.go:122] "RemoveContainer" containerID="c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd" Nov 02 15:39:06.051244 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.051219 5476 scope.go:122] "RemoveContainer" containerID="052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3" Nov 02 15:39:06.051592 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:06.051567 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3\": container with ID starting with 052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3 not found: ID does not exist" containerID="052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3" Nov 02 15:39:06.051714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.051601 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3"} err="failed to get container status \"052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3\": rpc error: code = NotFound desc = could not find container \"052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3\": container with ID starting with 052e1c76b4746ed2b17bca1d55178829c23eff1fcc9b7182f8f70f0ae86387f3 not found: ID does not exist" Nov 02 15:39:06.051714 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.051625 5476 scope.go:122] "RemoveContainer" containerID="c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd" Nov 02 15:39:06.051970 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:06.051927 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd\": container with ID starting with c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd not found: ID does not exist" containerID="c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd" Nov 02 15:39:06.052138 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.051979 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd"} err="failed to get container status \"c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd\": rpc error: code = NotFound desc = could not find container \"c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd\": container with ID starting with c77f36fd7fe980f27cddb06e14ca1232f6c4b35998786dc32911b6d2e7800dfd not found: ID does not exist" Nov 02 15:39:06.185200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.185168 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb"] Nov 02 15:39:06.192270 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.192203 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-69nbb"] Nov 02 15:39:06.302034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.301811 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz"] Nov 02 15:39:06.305826 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.305795 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-vtjvz"] Nov 02 15:39:06.408446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.408417 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss"] Nov 02 15:39:06.408603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:06.408461 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-pdmss"] Nov 02 15:39:07.923818 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:07.923787 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d61490-e5dd-4591-a3a8-2979db889bd8" path="/var/lib/kubelet/pods/77d61490-e5dd-4591-a3a8-2979db889bd8/volumes" Nov 02 15:39:07.924294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:07.924278 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" path="/var/lib/kubelet/pods/c3e6b2ff-a84f-428d-9df4-cb020df76d2f/volumes" Nov 02 15:39:07.924709 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:07.924695 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" path="/var/lib/kubelet/pods/fc1b9353-6505-4328-93c1-4e5aa6c2566a/volumes" Nov 02 15:39:08.817760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:08.817721 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" podUID="7f947d46-4501-4a17-ae94-72793390fd4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:39:08.817946 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:08.817778 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" podUID="9f01d442-c187-4005-88e0-e36e9cbabc5b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:39:10.990108 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:10.990070 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" containerID="cri-o://56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba" gracePeriod=45 Nov 02 15:39:11.002387 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.002356 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" containerID="cri-o://a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a" gracePeriod=45 Nov 02 15:39:11.267466 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.267441 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:39:11.296840 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.296812 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:39:11.428040 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.427870 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6215819d-1566-43c4-8c45-073d4756a048-tls-certs\" (UniqueName: \"kubernetes.io/secret/6215819d-1566-43c4-8c45-073d4756a048-tls-certs\") pod \"6215819d-1566-43c4-8c45-073d4756a048\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " Nov 02 15:39:11.428040 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.427944 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-tmp-dir\") pod \"6215819d-1566-43c4-8c45-073d4756a048\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " Nov 02 15:39:11.428040 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428021 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015490c4-4394-487f-90ff-15b5f189b583-kube-api-access-5s4gv\" (UniqueName: \"kubernetes.io/projected/015490c4-4394-487f-90ff-15b5f189b583-kube-api-access-5s4gv\") pod \"015490c4-4394-487f-90ff-15b5f189b583\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " Nov 02 15:39:11.428346 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428052 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-home\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-home\") pod \"6215819d-1566-43c4-8c45-073d4756a048\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " Nov 02 15:39:11.428346 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428089 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-model-cache\") pod \"015490c4-4394-487f-90ff-15b5f189b583\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " Nov 02 15:39:11.428346 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428152 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-tmp-dir\") pod \"015490c4-4394-487f-90ff-15b5f189b583\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " Nov 02 15:39:11.428346 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428212 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/015490c4-4394-487f-90ff-15b5f189b583-tls-certs\" (UniqueName: \"kubernetes.io/secret/015490c4-4394-487f-90ff-15b5f189b583-tls-certs\") pod \"015490c4-4394-487f-90ff-15b5f189b583\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " Nov 02 15:39:11.428346 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428313 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-dshm\") pod \"6215819d-1566-43c4-8c45-073d4756a048\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " Nov 02 15:39:11.428612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428345 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-tmp-dir" pod "6215819d-1566-43c4-8c45-073d4756a048" (UID: "6215819d-1566-43c4-8c45-073d4756a048"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.428612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428364 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6215819d-1566-43c4-8c45-073d4756a048-kube-api-access-9qt6p\" (UniqueName: \"kubernetes.io/projected/6215819d-1566-43c4-8c45-073d4756a048-kube-api-access-9qt6p\") pod \"6215819d-1566-43c4-8c45-073d4756a048\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " Nov 02 15:39:11.428612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428455 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-dshm\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-dshm\") pod \"015490c4-4394-487f-90ff-15b5f189b583\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " Nov 02 15:39:11.428612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428504 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-home\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-home\") pod \"015490c4-4394-487f-90ff-15b5f189b583\" (UID: \"015490c4-4394-487f-90ff-15b5f189b583\") " Nov 02 15:39:11.428612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428534 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-model-cache\") pod \"6215819d-1566-43c4-8c45-073d4756a048\" (UID: \"6215819d-1566-43c4-8c45-073d4756a048\") " Nov 02 15:39:11.428612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428541 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-model-cache" pod "015490c4-4394-487f-90ff-15b5f189b583" (UID: "015490c4-4394-487f-90ff-15b5f189b583"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.428920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428724 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-home" pod "6215819d-1566-43c4-8c45-073d4756a048" (UID: "6215819d-1566-43c4-8c45-073d4756a048"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.429520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.428985 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-model-cache" pod "6215819d-1566-43c4-8c45-073d4756a048" (UID: "6215819d-1566-43c4-8c45-073d4756a048"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.429520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.429215 5476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-tmp-dir\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.429520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.429235 5476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-home\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.429520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.429247 5476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-model-cache\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.429520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.429260 5476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-model-cache\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.429520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.429468 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-home" pod "015490c4-4394-487f-90ff-15b5f189b583" (UID: "015490c4-4394-487f-90ff-15b5f189b583"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.429956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.429656 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-tmp-dir" pod "015490c4-4394-487f-90ff-15b5f189b583" (UID: "015490c4-4394-487f-90ff-15b5f189b583"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.437352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.437303 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-dshm" pod "6215819d-1566-43c4-8c45-073d4756a048" (UID: "6215819d-1566-43c4-8c45-073d4756a048"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.437352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.437322 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-dshm" pod "015490c4-4394-487f-90ff-15b5f189b583" (UID: "015490c4-4394-487f-90ff-15b5f189b583"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.437352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.437321 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015490c4-4394-487f-90ff-15b5f189b583-kube-api-access-5s4gv" pod "015490c4-4394-487f-90ff-15b5f189b583" (UID: "015490c4-4394-487f-90ff-15b5f189b583"). InnerVolumeSpecName "kube-api-access-5s4gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:11.437352 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.437342 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6215819d-1566-43c4-8c45-073d4756a048-kube-api-access-9qt6p" pod "6215819d-1566-43c4-8c45-073d4756a048" (UID: "6215819d-1566-43c4-8c45-073d4756a048"). InnerVolumeSpecName "kube-api-access-9qt6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:11.437665 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.437396 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015490c4-4394-487f-90ff-15b5f189b583-tls-certs" pod "015490c4-4394-487f-90ff-15b5f189b583" (UID: "015490c4-4394-487f-90ff-15b5f189b583"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:11.438540 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.438515 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6215819d-1566-43c4-8c45-073d4756a048-tls-certs" pod "6215819d-1566-43c4-8c45-073d4756a048" (UID: "6215819d-1566-43c4-8c45-073d4756a048"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:11.531517 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.531433 5476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-home\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.531517 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.531475 5476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6215819d-1566-43c4-8c45-073d4756a048-tls-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.531517 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.531493 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s4gv\" (UniqueName: \"kubernetes.io/projected/015490c4-4394-487f-90ff-15b5f189b583-kube-api-access-5s4gv\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.531517 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.531509 5476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-tmp-dir\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.531768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.531522 5476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/015490c4-4394-487f-90ff-15b5f189b583-tls-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.531768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.531536 5476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6215819d-1566-43c4-8c45-073d4756a048-dshm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.531768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.531550 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qt6p\" (UniqueName: \"kubernetes.io/projected/6215819d-1566-43c4-8c45-073d4756a048-kube-api-access-9qt6p\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.531768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:11.531563 5476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/015490c4-4394-487f-90ff-15b5f189b583-dshm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:12.053819 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.053787 5476 generic.go:356] "Generic (PLEG): container finished" podID="6215819d-1566-43c4-8c45-073d4756a048" containerID="56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba" exitCode=0 Nov 02 15:39:12.054184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.053840 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" Nov 02 15:39:12.054184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.053868 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" event={"ID":"6215819d-1566-43c4-8c45-073d4756a048","Type":"ContainerDied","Data":"56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba"} Nov 02 15:39:12.054184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.053912 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8" event={"ID":"6215819d-1566-43c4-8c45-073d4756a048","Type":"ContainerDied","Data":"661f7e3b75ebde9c025a8bf42d7b02e2793786f1e34b66a31b5f4387692b0f2a"} Nov 02 15:39:12.054184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.053930 5476 scope.go:122] "RemoveContainer" containerID="56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba" Nov 02 15:39:12.055488 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.055469 5476 generic.go:356] "Generic (PLEG): container finished" podID="015490c4-4394-487f-90ff-15b5f189b583" containerID="a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a" exitCode=0 Nov 02 15:39:12.055586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.055530 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" event={"ID":"015490c4-4394-487f-90ff-15b5f189b583","Type":"ContainerDied","Data":"a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a"} Nov 02 15:39:12.055586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.055559 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" event={"ID":"015490c4-4394-487f-90ff-15b5f189b583","Type":"ContainerDied","Data":"ae4e1c12bf4894ff56a14cc417f7e44a9d48c3edea41fe3c443d75a5fc61f519"} Nov 02 15:39:12.055586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.055535 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf" Nov 02 15:39:12.064363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.064341 5476 scope.go:122] "RemoveContainer" containerID="56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba" Nov 02 15:39:12.064602 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:12.064582 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba\": container with ID starting with 56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba not found: ID does not exist" containerID="56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba" Nov 02 15:39:12.064649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.064610 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba"} err="failed to get container status \"56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba\": rpc error: code = NotFound desc = could not find container \"56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba\": container with ID starting with 56182d302fb0aa191cdb7fdaa67b9fb4f7f2feba643f5146e62dbeed324847ba not found: ID does not exist" Nov 02 15:39:12.064649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.064625 5476 scope.go:122] "RemoveContainer" containerID="a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a" Nov 02 15:39:12.073636 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.073616 5476 scope.go:122] "RemoveContainer" containerID="a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a" Nov 02 15:39:12.073890 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:12.073867 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a\": container with ID starting with a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a not found: ID does not exist" containerID="a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a" Nov 02 15:39:12.073939 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.073902 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a"} err="failed to get container status \"a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a\": rpc error: code = NotFound desc = could not find container \"a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a\": container with ID starting with a2cd249e5de90d64d1df0bf2c926425585c2e27d791d4f5d510796002b9bee6a not found: ID does not exist" Nov 02 15:39:12.210227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.210160 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf"] Nov 02 15:39:12.255966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.255939 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/premium-w3-71d1eb-kserve-7d7fb8f6b7-n4zcf"] Nov 02 15:39:12.316867 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.316839 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8"] Nov 02 15:39:12.321011 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:12.320965 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/distinct-w3-71d1eb-kserve-6cf679899d-mv2s8"] Nov 02 15:39:13.923808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:13.923773 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015490c4-4394-487f-90ff-15b5f189b583" path="/var/lib/kubelet/pods/015490c4-4394-487f-90ff-15b5f189b583/volumes" Nov 02 15:39:13.924281 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:13.924266 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6215819d-1566-43c4-8c45-073d4756a048" path="/var/lib/kubelet/pods/6215819d-1566-43c4-8c45-073d4756a048/volumes" Nov 02 15:39:18.817903 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:18.817867 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" podUID="9f01d442-c187-4005-88e0-e36e9cbabc5b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:39:18.818370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:18.817867 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" podUID="7f947d46-4501-4a17-ae94-72793390fd4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:39:20.148143 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:20.148108 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:39:21.016225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.016177 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:39:21.052383 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.052345 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f947d46-4501-4a17-ae94-72793390fd4f-kube-api-access-n2ptg\" (UniqueName: \"kubernetes.io/projected/7f947d46-4501-4a17-ae94-72793390fd4f-kube-api-access-n2ptg\") pod \"7f947d46-4501-4a17-ae94-72793390fd4f\" (UID: \"7f947d46-4501-4a17-ae94-72793390fd4f\") " Nov 02 15:39:21.052567 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.052547 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f947d46-4501-4a17-ae94-72793390fd4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f947d46-4501-4a17-ae94-72793390fd4f-plugins-config-volume\") pod \"7f947d46-4501-4a17-ae94-72793390fd4f\" (UID: \"7f947d46-4501-4a17-ae94-72793390fd4f\") " Nov 02 15:39:21.053171 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.053137 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f947d46-4501-4a17-ae94-72793390fd4f-plugins-config-volume" pod "7f947d46-4501-4a17-ae94-72793390fd4f" (UID: "7f947d46-4501-4a17-ae94-72793390fd4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:21.056269 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.056238 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f947d46-4501-4a17-ae94-72793390fd4f-kube-api-access-n2ptg" pod "7f947d46-4501-4a17-ae94-72793390fd4f" (UID: "7f947d46-4501-4a17-ae94-72793390fd4f"). InnerVolumeSpecName "kube-api-access-n2ptg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:21.090674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.090649 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:39:21.121199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.121163 5476 generic.go:356] "Generic (PLEG): container finished" podID="9f01d442-c187-4005-88e0-e36e9cbabc5b" containerID="f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9" exitCode=0 Nov 02 15:39:21.121385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.121213 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" Nov 02 15:39:21.121385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.121257 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" event={"ID":"9f01d442-c187-4005-88e0-e36e9cbabc5b","Type":"ContainerDied","Data":"f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9"} Nov 02 15:39:21.121385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.121341 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7" event={"ID":"9f01d442-c187-4005-88e0-e36e9cbabc5b","Type":"ContainerDied","Data":"f9ea63e40c3c3cd96d2538e4c45b3e015e3a084d55d430ba24e9aa67b527aa74"} Nov 02 15:39:21.121385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.121372 5476 scope.go:122] "RemoveContainer" containerID="f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9" Nov 02 15:39:21.127069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.126330 5476 generic.go:356] "Generic (PLEG): container finished" podID="7f947d46-4501-4a17-ae94-72793390fd4f" containerID="47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632" exitCode=0 Nov 02 15:39:21.127069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.126600 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" event={"ID":"7f947d46-4501-4a17-ae94-72793390fd4f","Type":"ContainerDied","Data":"47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632"} Nov 02 15:39:21.127069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.126648 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" event={"ID":"7f947d46-4501-4a17-ae94-72793390fd4f","Type":"ContainerDied","Data":"91891acf265b855ce5027df1263fc4c8c1c37d9ecdf940bd201b7272d03d0ae7"} Nov 02 15:39:21.127069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.126807 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf" Nov 02 15:39:21.136023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.135763 5476 scope.go:122] "RemoveContainer" containerID="f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9" Nov 02 15:39:21.136372 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:21.136338 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9\": container with ID starting with f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9 not found: ID does not exist" containerID="f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9" Nov 02 15:39:21.136489 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.136382 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9"} err="failed to get container status \"f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9\": rpc error: code = NotFound desc = could not find container \"f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9\": container with ID starting with f25bec8d503ff147704d71dac4cae02aa3626e92472071bbb1a13cc5b787f1d9 not found: ID does not exist" Nov 02 15:39:21.136489 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.136409 5476 scope.go:122] "RemoveContainer" containerID="47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632" Nov 02 15:39:21.148464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.148438 5476 scope.go:122] "RemoveContainer" containerID="47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632" Nov 02 15:39:21.148827 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:21.148803 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632\": container with ID starting with 47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632 not found: ID does not exist" containerID="47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632" Nov 02 15:39:21.148893 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.148839 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632"} err="failed to get container status \"47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632\": rpc error: code = NotFound desc = could not find container \"47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632\": container with ID starting with 47223f531e20d6721bf962012c8c847d9d4a57ec8f3e2943ad89aa53a64f9632 not found: ID does not exist" Nov 02 15:39:21.154328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.154303 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f01d442-c187-4005-88e0-e36e9cbabc5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f01d442-c187-4005-88e0-e36e9cbabc5b-plugins-config-volume\") pod \"9f01d442-c187-4005-88e0-e36e9cbabc5b\" (UID: \"9f01d442-c187-4005-88e0-e36e9cbabc5b\") " Nov 02 15:39:21.154499 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.154487 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f01d442-c187-4005-88e0-e36e9cbabc5b-kube-api-access-bkrzs\" (UniqueName: \"kubernetes.io/projected/9f01d442-c187-4005-88e0-e36e9cbabc5b-kube-api-access-bkrzs\") pod \"9f01d442-c187-4005-88e0-e36e9cbabc5b\" (UID: \"9f01d442-c187-4005-88e0-e36e9cbabc5b\") " Nov 02 15:39:21.154696 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.154670 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f01d442-c187-4005-88e0-e36e9cbabc5b-plugins-config-volume" pod "9f01d442-c187-4005-88e0-e36e9cbabc5b" (UID: "9f01d442-c187-4005-88e0-e36e9cbabc5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:21.154876 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.154856 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f01d442-c187-4005-88e0-e36e9cbabc5b-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:21.154922 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.154882 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f947d46-4501-4a17-ae94-72793390fd4f-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:21.154922 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.154897 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2ptg\" (UniqueName: \"kubernetes.io/projected/7f947d46-4501-4a17-ae94-72793390fd4f-kube-api-access-n2ptg\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:21.156479 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.156460 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f01d442-c187-4005-88e0-e36e9cbabc5b-kube-api-access-bkrzs" pod "9f01d442-c187-4005-88e0-e36e9cbabc5b" (UID: "9f01d442-c187-4005-88e0-e36e9cbabc5b"). InnerVolumeSpecName "kube-api-access-bkrzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:21.256170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.256083 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkrzs\" (UniqueName: \"kubernetes.io/projected/9f01d442-c187-4005-88e0-e36e9cbabc5b-kube-api-access-bkrzs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:21.404840 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.404800 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf"] Nov 02 15:39:21.410910 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.410871 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-61a835-555bdff7ff-mp2lf"] Nov 02 15:39:21.535368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.535284 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7"] Nov 02 15:39:21.539605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.539574 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-61a835-79bf7f5ddc-v9fd7"] Nov 02 15:39:21.924706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.924657 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f947d46-4501-4a17-ae94-72793390fd4f" path="/var/lib/kubelet/pods/7f947d46-4501-4a17-ae94-72793390fd4f/volumes" Nov 02 15:39:21.925221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:21.925200 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f01d442-c187-4005-88e0-e36e9cbabc5b" path="/var/lib/kubelet/pods/9f01d442-c187-4005-88e0-e36e9cbabc5b/volumes" Nov 02 15:39:31.585021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:31.583124 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:31.677780 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:31.675429 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj" podUID="4350e014-b37d-459a-a887-79d72ed3ed3c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:31.752018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:31.746140 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 02 15:39:31.752018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:31.751630 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:31.790041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:31.786083 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.925791 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh"] Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.926979 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927013 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927028 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerName="istio-proxy" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927034 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerName="istio-proxy" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927050 5476 cpu_manager.go:414] "containerMap: removing container" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927056 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927073 5476 cpu_manager.go:414] "containerMap: removing container" podUID="7f947d46-4501-4a17-ae94-72793390fd4f" containerName="payload-processing" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927078 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f947d46-4501-4a17-ae94-72793390fd4f" containerName="payload-processing" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927094 5476 cpu_manager.go:414] "containerMap: removing container" podUID="9a518198-bfea-4c7b-9696-fe657ab1cf90" containerName="maas-api" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927098 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a518198-bfea-4c7b-9696-fe657ab1cf90" containerName="maas-api" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927105 5476 cpu_manager.go:414] "containerMap: removing container" podUID="77d61490-e5dd-4591-a3a8-2979db889bd8" containerName="istio-proxy" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927109 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d61490-e5dd-4591-a3a8-2979db889bd8" containerName="istio-proxy" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927118 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927122 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927134 5476 cpu_manager.go:414] "containerMap: removing container" podUID="9f01d442-c187-4005-88e0-e36e9cbabc5b" containerName="payload-pre-processing" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927139 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f01d442-c187-4005-88e0-e36e9cbabc5b" containerName="payload-pre-processing" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927287 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927298 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1b9353-6505-4328-93c1-4e5aa6c2566a" containerName="istio-proxy" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927306 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="015490c4-4394-487f-90ff-15b5f189b583" containerName="main" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927313 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f947d46-4501-4a17-ae94-72793390fd4f" containerName="payload-processing" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927321 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a518198-bfea-4c7b-9696-fe657ab1cf90" containerName="maas-api" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927328 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d61490-e5dd-4591-a3a8-2979db889bd8" containerName="istio-proxy" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927334 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6215819d-1566-43c4-8c45-073d4756a048" containerName="main" Nov 02 15:39:35.928177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.927341 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f01d442-c187-4005-88e0-e36e9cbabc5b" containerName="payload-pre-processing" Nov 02 15:39:35.935251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.935223 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:35.962079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.962040 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-metrics-certs\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:35.962225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.962104 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b2c6\" (UniqueName: \"kubernetes.io/projected/7b128868-d225-48c1-bf70-b6d3d9d40ffc-kube-api-access-9b2c6\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:35.962225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.962211 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-maas-api-tls\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:35.962392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.962374 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b128868-d225-48c1-bf70-b6d3d9d40ffc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:35.992604 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.992271 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f6b245-serving-cert\"" Nov 02 15:39:35.995486 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:35.995456 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh"] Nov 02 15:39:36.064249 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:36.064208 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b128868-d225-48c1-bf70-b6d3d9d40ffc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:36.064448 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:36.064326 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-metrics-certs\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:36.064448 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:36.064359 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b2c6\" (UniqueName: \"kubernetes.io/projected/7b128868-d225-48c1-bf70-b6d3d9d40ffc-kube-api-access-9b2c6\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:36.064587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:36.064488 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-maas-api-tls\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:36.064719 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:36.064699 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-serving-cert: secret "maas-api-e2e-inf-b-f6b245-serving-cert" not found Nov 02 15:39:36.065125 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:36.065102 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-maas-api-tls podName:7b128868-d225-48c1-bf70-b6d3d9d40ffc nodeName:}" failed. No retries permitted until 2026-11-02 15:39:36.565076786 +0000 UTC m=+2478.767954826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-maas-api-tls") pod "maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" (UID: "7b128868-d225-48c1-bf70-b6d3d9d40ffc") : secret "maas-api-e2e-inf-b-f6b245-serving-cert" not found Nov 02 15:39:36.067897 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:36.067867 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-metrics-certs\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:36.069361 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:36.069335 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b128868-d225-48c1-bf70-b6d3d9d40ffc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:36.122325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:36.122281 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b2c6\" (UniqueName: \"kubernetes.io/projected/7b128868-d225-48c1-bf70-b6d3d9d40ffc-kube-api-access-9b2c6\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:36.572755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:36.572721 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-maas-api-tls\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:36.575370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:36.575335 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-maas-api-tls\") pod \"maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:36.855762 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:36.855737 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:37.240305 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:37.240208 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh"] Nov 02 15:39:37.246155 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:39:37.246123 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b128868_d225_48c1_bf70_b6d3d9d40ffc.slice/crio-954ba15ab34a2ac1d76053be4e34f5933b76c03dd2971ec04e75271a33b12784 WatchSource:0}: Error finding container 954ba15ab34a2ac1d76053be4e34f5933b76c03dd2971ec04e75271a33b12784: Status 404 returned error can't find the container with id 954ba15ab34a2ac1d76053be4e34f5933b76c03dd2971ec04e75271a33b12784 Nov 02 15:39:37.351252 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:37.351209 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" event={"ID":"7b128868-d225-48c1-bf70-b6d3d9d40ffc","Type":"ContainerStarted","Data":"954ba15ab34a2ac1d76053be4e34f5933b76c03dd2971ec04e75271a33b12784"} Nov 02 15:39:38.296203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:38.296146 5476 patch_prober.go:29] interesting pod/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:39:38.296700 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:38.296222 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:39:38.362488 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:38.362443 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" event={"ID":"7b128868-d225-48c1-bf70-b6d3d9d40ffc","Type":"ContainerStarted","Data":"e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750"} Nov 02 15:39:38.362690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:38.362571 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:38.524890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:38.524773 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" podStartSLOduration=2.618906776 podStartE2EDuration="3.524753404s" totalImagesPullingTime="905.846628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:38.521698944 +0000 UTC m=+2480.724577003" watchObservedRunningTime="2026-11-02 15:39:38.524753404 +0000 UTC m=+2480.727631465" Nov 02 15:39:41.390590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:41.390553 5476 generic.go:356] "Generic (PLEG): container finished" podID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerID="726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00" exitCode=0 Nov 02 15:39:41.391108 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:41.390640 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" event={"ID":"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b","Type":"ContainerDied","Data":"726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00"} Nov 02 15:39:41.391199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:41.391175 5476 scope.go:122] "RemoveContainer" containerID="726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00" Nov 02 15:39:41.393055 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:41.393025 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:41.393134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:41.393117 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:41.393185 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:41.393169 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:42.403671 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:42.403640 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" event={"ID":"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b","Type":"ContainerStarted","Data":"ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7"} Nov 02 15:39:43.124221 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:43.124187 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:39:43.124463 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:43.124437 5476 patch_prober.go:29] interesting pod/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 02 15:39:43.124522 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:43.124498 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 02 15:39:44.124619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:44.124587 5476 patch_prober.go:29] interesting pod/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 02 15:39:44.125031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:44.124638 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 02 15:39:44.397775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:44.397697 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:39:45.124678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:45.124640 5476 patch_prober.go:29] interesting pod/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 02 15:39:45.125100 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:45.124697 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 02 15:39:46.124765 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:46.124723 5476 patch_prober.go:29] interesting pod/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 02 15:39:46.125291 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:46.124781 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 02 15:39:47.124138 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:47.124103 5476 patch_prober.go:29] interesting pod/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Nov 02 15:39:47.124326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:47.124153 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Nov 02 15:39:48.127958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:48.127929 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:39:48.128401 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:48.128343 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:39:48.128930 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:48.128913 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:39:49.432854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.432820 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7"] Nov 02 15:39:49.433490 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.433470 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" Nov 02 15:39:49.433490 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.433489 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" Nov 02 15:39:49.433649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.433605 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e6b2ff-a84f-428d-9df4-cb020df76d2f" containerName="istio-proxy" Nov 02 15:39:49.437233 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.437216 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.454146 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.454115 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7"] Nov 02 15:39:49.494322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.494292 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-71d1eb-openshift-default-dockercfg-zn6lk\"" Nov 02 15:39:49.507041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.506984 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.507187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.507148 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.507187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.507183 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.507371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.507211 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.507371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.507336 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.507371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.507368 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c634c862-ab4e-4e94-b08a-aa60620afc31-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.507558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.507394 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k98lx\" (UniqueName: \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-kube-api-access-k98lx\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.507558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.507512 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.507558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.507544 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609068 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609027 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609091 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c634c862-ab4e-4e94-b08a-aa60620afc31-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609127 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k98lx\" (UniqueName: \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-kube-api-access-k98lx\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609206 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609419 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609246 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609419 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609367 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609535 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609490 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609535 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609510 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609542 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609577 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609581 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609887 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609866 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609902 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.609977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.609946 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c634c862-ab4e-4e94-b08a-aa60620afc31-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.611584 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.611557 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.611893 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.611876 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.666978 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.666945 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.667150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.667106 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k98lx\" (UniqueName: \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-kube-api-access-k98lx\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:49.752337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:49.752241 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:50.146586 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:39:50.146548 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:39:50.182364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:50.182332 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7"] Nov 02 15:39:50.186026 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:39:50.185985 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc634c862_ab4e_4e94_b08a_aa60620afc31.slice/crio-9800b2d4246584ff7cdedfc345bcc81f7cc20a5ce78974299b926f81052e4243 WatchSource:0}: Error finding container 9800b2d4246584ff7cdedfc345bcc81f7cc20a5ce78974299b926f81052e4243: Status 404 returned error can't find the container with id 9800b2d4246584ff7cdedfc345bcc81f7cc20a5ce78974299b926f81052e4243 Nov 02 15:39:50.188581 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:50.188548 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:50.188657 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:50.188618 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:50.188657 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:50.188644 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:50.467653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:50.467566 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" event={"ID":"c634c862-ab4e-4e94-b08a-aa60620afc31","Type":"ContainerStarted","Data":"eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9"} Nov 02 15:39:50.467653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:50.467634 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" event={"ID":"c634c862-ab4e-4e94-b08a-aa60620afc31","Type":"ContainerStarted","Data":"9800b2d4246584ff7cdedfc345bcc81f7cc20a5ce78974299b926f81052e4243"} Nov 02 15:39:50.585909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:50.585851 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" podStartSLOduration=1.585834496 podStartE2EDuration="1.585834496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:50.580382064 +0000 UTC m=+2492.783260135" watchObservedRunningTime="2026-11-02 15:39:50.585834496 +0000 UTC m=+2492.788712559" Nov 02 15:39:50.752415 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:50.752333 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:51.755180 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:51.755148 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 15:39:51.755600 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:51.755204 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 15:39:52.753567 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:52.753530 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 15:39:52.753743 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:52.753592 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 15:39:53.752971 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:53.752930 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 15:39:53.753477 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:53.753018 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 15:39:54.753189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:54.753158 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 15:39:54.753956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:54.753209 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 15:39:55.755538 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:55.755504 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 15:39:55.756214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:55.756177 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 15:39:56.753307 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:56.753277 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Nov 02 15:39:56.753485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:56.753330 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Nov 02 15:39:57.761858 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:57.761814 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:57.762623 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:57.762584 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:39:57.766417 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:39:57.766394 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:40:20.147758 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:40:20.147712 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:40:22.200465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:22.200423 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:40:25.385771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:25.385734 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7"] Nov 02 15:40:25.386232 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:25.386037 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerName="istio-proxy" containerID="cri-o://eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9" gracePeriod=30 Nov 02 15:40:27.765215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:27.765132 5476 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:40:27.765215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:27.765203 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:40:30.669092 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.669028 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:40:30.703771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.703732 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-token\" (UniqueName: \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-token\") pod \"c634c862-ab4e-4e94-b08a-aa60620afc31\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " Nov 02 15:40:30.706802 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.704069 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-data\") pod \"c634c862-ab4e-4e94-b08a-aa60620afc31\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " Nov 02 15:40:30.707182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.707148 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-podinfo\") pod \"c634c862-ab4e-4e94-b08a-aa60620afc31\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " Nov 02 15:40:30.707646 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.707625 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-socket\") pod \"c634c862-ab4e-4e94-b08a-aa60620afc31\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " Nov 02 15:40:30.708034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.707975 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-credential-socket\") pod \"c634c862-ab4e-4e94-b08a-aa60620afc31\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " Nov 02 15:40:30.708320 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.708293 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-envoy\") pod \"c634c862-ab4e-4e94-b08a-aa60620afc31\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " Nov 02 15:40:30.708430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.708402 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-certs\") pod \"c634c862-ab4e-4e94-b08a-aa60620afc31\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " Nov 02 15:40:30.708501 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.708464 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-kube-api-access-k98lx\" (UniqueName: \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-kube-api-access-k98lx\") pod \"c634c862-ab4e-4e94-b08a-aa60620afc31\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " Nov 02 15:40:30.708563 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.708520 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c634c862-ab4e-4e94-b08a-aa60620afc31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c634c862-ab4e-4e94-b08a-aa60620afc31-istiod-ca-cert\") pod \"c634c862-ab4e-4e94-b08a-aa60620afc31\" (UID: \"c634c862-ab4e-4e94-b08a-aa60620afc31\") " Nov 02 15:40:30.710078 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.704537 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-data" pod "c634c862-ab4e-4e94-b08a-aa60620afc31" (UID: "c634c862-ab4e-4e94-b08a-aa60620afc31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.710247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.710050 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c634c862-ab4e-4e94-b08a-aa60620afc31-istiod-ca-cert" pod "c634c862-ab4e-4e94-b08a-aa60620afc31" (UID: "c634c862-ab4e-4e94-b08a-aa60620afc31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:40:30.713170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.713142 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-certs" pod "c634c862-ab4e-4e94-b08a-aa60620afc31" (UID: "c634c862-ab4e-4e94-b08a-aa60620afc31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.713661 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.713512 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-envoy" pod "c634c862-ab4e-4e94-b08a-aa60620afc31" (UID: "c634c862-ab4e-4e94-b08a-aa60620afc31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.713833 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.713762 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-credential-socket" pod "c634c862-ab4e-4e94-b08a-aa60620afc31" (UID: "c634c862-ab4e-4e94-b08a-aa60620afc31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.713833 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.713799 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-socket" pod "c634c862-ab4e-4e94-b08a-aa60620afc31" (UID: "c634c862-ab4e-4e94-b08a-aa60620afc31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.715175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.715150 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-podinfo" pod "c634c862-ab4e-4e94-b08a-aa60620afc31" (UID: "c634c862-ab4e-4e94-b08a-aa60620afc31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:40:30.716146 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.716122 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-kube-api-access-k98lx" pod "c634c862-ab4e-4e94-b08a-aa60620afc31" (UID: "c634c862-ab4e-4e94-b08a-aa60620afc31"). InnerVolumeSpecName "kube-api-access-k98lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:30.716385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.716363 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-token" pod "c634c862-ab4e-4e94-b08a-aa60620afc31" (UID: "c634c862-ab4e-4e94-b08a-aa60620afc31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:30.793079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.793043 5476 generic.go:356] "Generic (PLEG): container finished" podID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerID="eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9" exitCode=0 Nov 02 15:40:30.793214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.793101 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" Nov 02 15:40:30.793214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.793136 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" event={"ID":"c634c862-ab4e-4e94-b08a-aa60620afc31","Type":"ContainerDied","Data":"eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9"} Nov 02 15:40:30.793214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.793194 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7" event={"ID":"c634c862-ab4e-4e94-b08a-aa60620afc31","Type":"ContainerDied","Data":"9800b2d4246584ff7cdedfc345bcc81f7cc20a5ce78974299b926f81052e4243"} Nov 02 15:40:30.793347 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.793224 5476 scope.go:122] "RemoveContainer" containerID="eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9" Nov 02 15:40:30.807566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.807546 5476 scope.go:122] "RemoveContainer" containerID="eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9" Nov 02 15:40:30.807866 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:40:30.807846 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9\": container with ID starting with eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9 not found: ID does not exist" containerID="eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9" Nov 02 15:40:30.807935 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.807879 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9"} err="failed to get container status \"eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9\": rpc error: code = NotFound desc = could not find container \"eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9\": container with ID starting with eb2ec9d1610b87d8d3fa5596c2e4b9ab89a110d0caa309792b7fefa9d57779e9 not found: ID does not exist" Nov 02 15:40:30.810669 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.810650 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.810733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.810672 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k98lx\" (UniqueName: \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-kube-api-access-k98lx\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.810733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.810684 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c634c862-ab4e-4e94-b08a-aa60620afc31-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.810733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.810694 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.810733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.810703 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.810733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.810711 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.810733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.810720 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.810733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.810728 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.811003 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.810736 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c634c862-ab4e-4e94-b08a-aa60620afc31-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:30.949508 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.949400 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7"] Nov 02 15:40:30.953662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:30.953638 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-t5rc7"] Nov 02 15:40:31.923253 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:31.923215 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" path="/var/lib/kubelet/pods/c634c862-ab4e-4e94-b08a-aa60620afc31/volumes" Nov 02 15:40:50.149616 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:40:50.149578 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:40:55.985326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:40:55.985161 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl"] Nov 02 15:41:03.055963 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:03.055930 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:41:05.645132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:05.645097 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kk2m9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:41:05.645614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:05.645174 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podUID="afd13c1e-9b91-4be5-92ed-ceea14f0203c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:41:11.017817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.017774 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" containerID="cri-o://5f79631293758e9c79dbb8a9782c95632690fe3a63131dfedba7881047c36414" gracePeriod=45 Nov 02 15:41:11.194756 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.194714 5476 generic.go:356] "Generic (PLEG): container finished" podID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerID="5f79631293758e9c79dbb8a9782c95632690fe3a63131dfedba7881047c36414" exitCode=0 Nov 02 15:41:11.194941 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.194920 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" event={"ID":"50036e4d-3d5c-4bb4-8ea8-98193d52cb16","Type":"ContainerDied","Data":"5f79631293758e9c79dbb8a9782c95632690fe3a63131dfedba7881047c36414"} Nov 02 15:41:11.280550 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.280486 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:41:11.325865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.325837 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tls-certs\" (UniqueName: \"kubernetes.io/secret/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tls-certs\") pod \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " Nov 02 15:41:11.326020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.325907 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-model-cache\") pod \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " Nov 02 15:41:11.326020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.325931 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tmp-dir\") pod \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " Nov 02 15:41:11.326020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.325958 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-home\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-home\") pod \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " Nov 02 15:41:11.326201 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.326029 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-kube-api-access-rgvcq\" (UniqueName: \"kubernetes.io/projected/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-kube-api-access-rgvcq\") pod \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " Nov 02 15:41:11.326201 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.326138 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-dshm\") pod \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\" (UID: \"50036e4d-3d5c-4bb4-8ea8-98193d52cb16\") " Nov 02 15:41:11.326304 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.326182 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-model-cache" pod "50036e4d-3d5c-4bb4-8ea8-98193d52cb16" (UID: "50036e4d-3d5c-4bb4-8ea8-98193d52cb16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.326304 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.326223 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tmp-dir" pod "50036e4d-3d5c-4bb4-8ea8-98193d52cb16" (UID: "50036e4d-3d5c-4bb4-8ea8-98193d52cb16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.326393 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.326299 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-home" pod "50036e4d-3d5c-4bb4-8ea8-98193d52cb16" (UID: "50036e4d-3d5c-4bb4-8ea8-98193d52cb16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.326651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.326630 5476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-model-cache\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.326707 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.326682 5476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tmp-dir\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.326707 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.326699 5476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-home\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.328187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.328163 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tls-certs" pod "50036e4d-3d5c-4bb4-8ea8-98193d52cb16" (UID: "50036e4d-3d5c-4bb4-8ea8-98193d52cb16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:41:11.328187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.328177 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-dshm" pod "50036e4d-3d5c-4bb4-8ea8-98193d52cb16" (UID: "50036e4d-3d5c-4bb4-8ea8-98193d52cb16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.328318 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.328189 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-kube-api-access-rgvcq" pod "50036e4d-3d5c-4bb4-8ea8-98193d52cb16" (UID: "50036e4d-3d5c-4bb4-8ea8-98193d52cb16"). InnerVolumeSpecName "kube-api-access-rgvcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:11.427710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.427680 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgvcq\" (UniqueName: \"kubernetes.io/projected/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-kube-api-access-rgvcq\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.427710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.427706 5476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-dshm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.427710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:11.427715 5476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50036e4d-3d5c-4bb4-8ea8-98193d52cb16-tls-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:12.203231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:12.203152 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" Nov 02 15:41:12.203231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:12.203194 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl" event={"ID":"50036e4d-3d5c-4bb4-8ea8-98193d52cb16","Type":"ContainerDied","Data":"b035191a067d116d92b29e2cfe8c69d736ad6422cf24c95b60a4b5e23b967f5e"} Nov 02 15:41:12.203692 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:12.203271 5476 scope.go:122] "RemoveContainer" containerID="5f79631293758e9c79dbb8a9782c95632690fe3a63131dfedba7881047c36414" Nov 02 15:41:12.360790 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:12.360756 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl"] Nov 02 15:41:12.407067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:12.407038 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-891723/distinct-2-w5-891723-kserve-557bc4b587-nltfl"] Nov 02 15:41:13.922801 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:13.922768 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" path="/var/lib/kubelet/pods/50036e4d-3d5c-4bb4-8ea8-98193d52cb16/volumes" Nov 02 15:41:16.219757 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:16.219722 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq"] Nov 02 15:41:16.220617 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:16.220582 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" containerID="cri-o://0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865" gracePeriod=30 Nov 02 15:41:19.465750 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:19.465706 5476 scope.go:122] "RemoveContainer" containerID="8517e57ce9747b80e7ec258330edc2eee0f25535cdc57a21de18663dd271c7db" Nov 02 15:41:19.475664 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:19.475641 5476 scope.go:122] "RemoveContainer" containerID="e30f0496023ded410a1bb27f3e1fa769b71c0e134e002c49e0793cd7218926e4" Nov 02 15:41:20.147380 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:41:20.147343 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:41:20.594603 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:20.594505 5476 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:41:20.594961 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:20.594590 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:41:21.487893 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.487819 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667049 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-data\") pod \"037fbba8-633a-4c23-bd5e-d138c3c85732\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667102 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037fbba8-633a-4c23-bd5e-d138c3c85732-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037fbba8-633a-4c23-bd5e-d138c3c85732-istiod-ca-cert\") pod \"037fbba8-633a-4c23-bd5e-d138c3c85732\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667267 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-certs\") pod \"037fbba8-633a-4c23-bd5e-d138c3c85732\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667354 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-credential-socket\") pod \"037fbba8-633a-4c23-bd5e-d138c3c85732\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667403 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-socket\") pod \"037fbba8-633a-4c23-bd5e-d138c3c85732\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667460 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-kube-api-access-wngbc\" (UniqueName: \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-kube-api-access-wngbc\") pod \"037fbba8-633a-4c23-bd5e-d138c3c85732\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667495 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-envoy\") pod \"037fbba8-633a-4c23-bd5e-d138c3c85732\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667526 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-podinfo\") pod \"037fbba8-633a-4c23-bd5e-d138c3c85732\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667557 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-token\" (UniqueName: \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-token\") pod \"037fbba8-633a-4c23-bd5e-d138c3c85732\" (UID: \"037fbba8-633a-4c23-bd5e-d138c3c85732\") " Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667685 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-credential-socket" pod "037fbba8-633a-4c23-bd5e-d138c3c85732" (UID: "037fbba8-633a-4c23-bd5e-d138c3c85732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667760 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037fbba8-633a-4c23-bd5e-d138c3c85732-istiod-ca-cert" pod "037fbba8-633a-4c23-bd5e-d138c3c85732" (UID: "037fbba8-633a-4c23-bd5e-d138c3c85732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:41:21.668107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.667860 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-data" pod "037fbba8-633a-4c23-bd5e-d138c3c85732" (UID: "037fbba8-633a-4c23-bd5e-d138c3c85732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:21.669290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.668286 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.669290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.668307 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/037fbba8-633a-4c23-bd5e-d138c3c85732-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.669290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.668324 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.669290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.668686 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-certs" pod "037fbba8-633a-4c23-bd5e-d138c3c85732" (UID: "037fbba8-633a-4c23-bd5e-d138c3c85732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:21.669290 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.668861 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-socket" pod "037fbba8-633a-4c23-bd5e-d138c3c85732" (UID: "037fbba8-633a-4c23-bd5e-d138c3c85732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:21.671276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.671238 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-kube-api-access-wngbc" pod "037fbba8-633a-4c23-bd5e-d138c3c85732" (UID: "037fbba8-633a-4c23-bd5e-d138c3c85732"). InnerVolumeSpecName "kube-api-access-wngbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:21.673178 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.672548 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-podinfo" pod "037fbba8-633a-4c23-bd5e-d138c3c85732" (UID: "037fbba8-633a-4c23-bd5e-d138c3c85732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:41:21.673178 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.673143 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-envoy" pod "037fbba8-633a-4c23-bd5e-d138c3c85732" (UID: "037fbba8-633a-4c23-bd5e-d138c3c85732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:21.673705 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.673678 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-token" pod "037fbba8-633a-4c23-bd5e-d138c3c85732" (UID: "037fbba8-633a-4c23-bd5e-d138c3c85732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:21.782943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.782854 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.783195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.783178 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.783313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.783301 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.783414 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.783401 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wngbc\" (UniqueName: \"kubernetes.io/projected/037fbba8-633a-4c23-bd5e-d138c3c85732-kube-api-access-wngbc\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.783527 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.783515 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.783628 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:21.783618 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/037fbba8-633a-4c23-bd5e-d138c3c85732-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:22.276938 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.276907 5476 generic.go:356] "Generic (PLEG): container finished" podID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerID="0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865" exitCode=0 Nov 02 15:41:22.277132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.276950 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" Nov 02 15:41:22.277132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.276963 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" event={"ID":"037fbba8-633a-4c23-bd5e-d138c3c85732","Type":"ContainerDied","Data":"0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865"} Nov 02 15:41:22.277132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.277036 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq" event={"ID":"037fbba8-633a-4c23-bd5e-d138c3c85732","Type":"ContainerDied","Data":"642bc4e2f05b4ae637f5d27555772cafaad8f8bdca1443b236eec757cddb12ab"} Nov 02 15:41:22.277132 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.277066 5476 scope.go:122] "RemoveContainer" containerID="0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865" Nov 02 15:41:22.289195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.287974 5476 scope.go:122] "RemoveContainer" containerID="18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52" Nov 02 15:41:22.297152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.297133 5476 scope.go:122] "RemoveContainer" containerID="0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865" Nov 02 15:41:22.297400 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:41:22.297379 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865\": container with ID starting with 0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865 not found: ID does not exist" containerID="0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865" Nov 02 15:41:22.297475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.297410 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865"} err="failed to get container status \"0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865\": rpc error: code = NotFound desc = could not find container \"0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865\": container with ID starting with 0820a0192afc8dd627c6b30c5a8be8a63db8f3ef942f12d0ded3356ec7a36865 not found: ID does not exist" Nov 02 15:41:22.297475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.297434 5476 scope.go:122] "RemoveContainer" containerID="18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52" Nov 02 15:41:22.297683 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:41:22.297667 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52\": container with ID starting with 18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52 not found: ID does not exist" containerID="18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52" Nov 02 15:41:22.297727 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.297690 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52"} err="failed to get container status \"18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52\": rpc error: code = NotFound desc = could not find container \"18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52\": container with ID starting with 18b1ecfff1553d2f207cb1fd99cac4980208f37665e146a075d28f0e66808d52 not found: ID does not exist" Nov 02 15:41:22.481133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.481101 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq"] Nov 02 15:41:22.494930 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:22.494901 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-cvdfq"] Nov 02 15:41:23.923468 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:23.923435 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" path="/var/lib/kubelet/pods/037fbba8-633a-4c23-bd5e-d138c3c85732/volumes" Nov 02 15:41:46.808565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:41:46.808521 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:41:50.147138 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:41:50.147103 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:42:19.514085 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:19.514051 5476 scope.go:122] "RemoveContainer" containerID="d09395bcb7005d3fa543fa07eb34e98a473e4538c67b91c8ba8941a3ff5cad93" Nov 02 15:42:20.147584 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:42:20.147547 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:42:37.293557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:37.293518 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw"] Nov 02 15:42:37.294124 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:37.293753 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" containerID="cri-o://f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c" gracePeriod=30 Nov 02 15:42:38.264036 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:38.263984 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:42:42.293873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.293837 5476 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:42:42.294294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.293900 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:42:42.583364 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.583331 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:42:42.643358 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643322 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-kube-api-access-mnphw\" (UniqueName: \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-kube-api-access-mnphw\") pod \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " Nov 02 15:42:42.643550 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643387 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istiod-ca-cert\") pod \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " Nov 02 15:42:42.643550 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643423 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-data\") pod \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " Nov 02 15:42:42.643550 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643505 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-credential-socket\") pod \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " Nov 02 15:42:42.643732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643569 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-certs\") pod \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " Nov 02 15:42:42.643732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643588 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-token\" (UniqueName: \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-token\") pod \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " Nov 02 15:42:42.643732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643638 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-socket\") pod \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " Nov 02 15:42:42.643732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643690 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-podinfo\") pod \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " Nov 02 15:42:42.643732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643719 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-envoy\") pod \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\" (UID: \"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294\") " Nov 02 15:42:42.644015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643816 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-data" pod "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" (UID: "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:42:42.644015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643881 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-credential-socket" pod "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" (UID: "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:42:42.644015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643941 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istiod-ca-cert" pod "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" (UID: "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:42:42.644015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.643982 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-socket" pod "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" (UID: "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:42:42.644261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.644181 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-certs" pod "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" (UID: "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:42:42.644389 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.644367 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:42.644460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.644392 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:42.644460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.644402 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:42.644460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.644410 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:42.644460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.644418 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:42.645745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.645716 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-kube-api-access-mnphw" pod "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" (UID: "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294"). InnerVolumeSpecName "kube-api-access-mnphw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:42:42.645988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.645967 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-token" pod "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" (UID: "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:42:42.646323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.646300 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-envoy" pod "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" (UID: "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:42:42.646434 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.646394 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-podinfo" pod "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" (UID: "5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:42:42.745978 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.745922 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:42.745978 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.745978 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:42.746248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.746008 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnphw\" (UniqueName: \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-kube-api-access-mnphw\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:42.746248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.746026 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:42.849466 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.849375 5476 generic.go:356] "Generic (PLEG): container finished" podID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerID="f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c" exitCode=0 Nov 02 15:42:42.849466 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.849424 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" Nov 02 15:42:42.849466 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.849458 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" event={"ID":"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294","Type":"ContainerDied","Data":"f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c"} Nov 02 15:42:42.849745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.849545 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw" event={"ID":"5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294","Type":"ContainerDied","Data":"a15ca2381eb07959df570eb5716511791c3e8b1446b9fde1ae99c2e79c180b6d"} Nov 02 15:42:42.849745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.849573 5476 scope.go:122] "RemoveContainer" containerID="f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c" Nov 02 15:42:42.863599 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.863468 5476 scope.go:122] "RemoveContainer" containerID="992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad" Nov 02 15:42:42.875914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.875889 5476 scope.go:122] "RemoveContainer" containerID="f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c" Nov 02 15:42:42.876324 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:42:42.876303 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c\": container with ID starting with f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c not found: ID does not exist" containerID="f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c" Nov 02 15:42:42.876416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.876333 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c"} err="failed to get container status \"f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c\": rpc error: code = NotFound desc = could not find container \"f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c\": container with ID starting with f758620054921a90f9d5e766af2a374660105b00e1194227798e0cbdfc7b787c not found: ID does not exist" Nov 02 15:42:42.876416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.876357 5476 scope.go:122] "RemoveContainer" containerID="992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad" Nov 02 15:42:42.876692 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:42:42.876668 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad\": container with ID starting with 992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad not found: ID does not exist" containerID="992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad" Nov 02 15:42:42.876840 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:42.876696 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad"} err="failed to get container status \"992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad\": rpc error: code = NotFound desc = could not find container \"992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad\": container with ID starting with 992adae210b670c4c8da8ca194a629321bccafb1f666229c279a916fc82f8cad not found: ID does not exist" Nov 02 15:42:43.006127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:43.006087 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw"] Nov 02 15:42:43.014202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:43.014167 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-7czrw"] Nov 02 15:42:43.923432 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:43.923399 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" path="/var/lib/kubelet/pods/5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294/volumes" Nov 02 15:42:50.146827 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:42:50.146788 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:42:55.315405 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:42:55.315369 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2"] Nov 02 15:43:00.316628 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.316586 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" podUID="9d123955-d5a6-4b5c-9f09-62fe07a5ddb5" containerName="payload-pre-processing" containerID="cri-o://f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759" gracePeriod=25 Nov 02 15:43:00.592694 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.592660 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:43:00.761472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.761438 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-plugins-config-volume\") pod \"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5\" (UID: \"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5\") " Nov 02 15:43:00.761677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.761578 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-kube-api-access-64cj8\" (UniqueName: \"kubernetes.io/projected/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-kube-api-access-64cj8\") pod \"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5\" (UID: \"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5\") " Nov 02 15:43:00.761888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.761861 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-plugins-config-volume" pod "9d123955-d5a6-4b5c-9f09-62fe07a5ddb5" (UID: "9d123955-d5a6-4b5c-9f09-62fe07a5ddb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:43:00.762086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.762070 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:00.763670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.763643 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-kube-api-access-64cj8" pod "9d123955-d5a6-4b5c-9f09-62fe07a5ddb5" (UID: "9d123955-d5a6-4b5c-9f09-62fe07a5ddb5"). InnerVolumeSpecName "kube-api-access-64cj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:00.862961 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.862932 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64cj8\" (UniqueName: \"kubernetes.io/projected/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5-kube-api-access-64cj8\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:00.967956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.967925 5476 generic.go:356] "Generic (PLEG): container finished" podID="9d123955-d5a6-4b5c-9f09-62fe07a5ddb5" containerID="f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759" exitCode=0 Nov 02 15:43:00.968162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.967977 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" Nov 02 15:43:00.968162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.967988 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" event={"ID":"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5","Type":"ContainerDied","Data":"f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759"} Nov 02 15:43:00.968162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.968068 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2" event={"ID":"9d123955-d5a6-4b5c-9f09-62fe07a5ddb5","Type":"ContainerDied","Data":"2c754361110d8823134d7036c98db3bc7ae95f0745fc60070f091bebabb3f809"} Nov 02 15:43:00.968162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.968090 5476 scope.go:122] "RemoveContainer" containerID="f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759" Nov 02 15:43:00.978332 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.978313 5476 scope.go:122] "RemoveContainer" containerID="f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759" Nov 02 15:43:00.978637 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:43:00.978615 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759\": container with ID starting with f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759 not found: ID does not exist" containerID="f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759" Nov 02 15:43:00.978691 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:00.978644 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759"} err="failed to get container status \"f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759\": rpc error: code = NotFound desc = could not find container \"f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759\": container with ID starting with f0f38cc8bd536faee0e26689ba51eed31f8e305405301937b787f2af1d624759 not found: ID does not exist" Nov 02 15:43:01.124169 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:01.124081 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2"] Nov 02 15:43:01.169209 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:01.169182 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c35ee-864459cd87-g5fw2"] Nov 02 15:43:01.925106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:01.925070 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d123955-d5a6-4b5c-9f09-62fe07a5ddb5" path="/var/lib/kubelet/pods/9d123955-d5a6-4b5c-9f09-62fe07a5ddb5/volumes" Nov 02 15:43:11.549281 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:11.549249 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:43:18.613702 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:18.613672 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:43:18.616570 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:18.616549 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:43:20.147356 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:43:20.147313 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:43:33.185466 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:33.185391 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb"] Nov 02 15:43:33.290486 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:33.290456 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq"] Nov 02 15:43:38.186152 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.186090 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" podUID="e9223358-7b05-4518-a90f-e12bbaf5c473" containerName="payload-processing" containerID="cri-o://ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746" gracePeriod=25 Nov 02 15:43:38.291377 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.291334 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" podUID="f62a8958-cbe4-431c-aaa6-e2644e34007f" containerName="payload-pre-processing" containerID="cri-o://8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d" gracePeriod=25 Nov 02 15:43:38.505600 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.505570 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:43:38.564087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.564064 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:43:38.650987 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.650954 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9223358-7b05-4518-a90f-e12bbaf5c473-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9223358-7b05-4518-a90f-e12bbaf5c473-plugins-config-volume\") pod \"e9223358-7b05-4518-a90f-e12bbaf5c473\" (UID: \"e9223358-7b05-4518-a90f-e12bbaf5c473\") " Nov 02 15:43:38.651184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.651066 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9223358-7b05-4518-a90f-e12bbaf5c473-kube-api-access-v9g6m\" (UniqueName: \"kubernetes.io/projected/e9223358-7b05-4518-a90f-e12bbaf5c473-kube-api-access-v9g6m\") pod \"e9223358-7b05-4518-a90f-e12bbaf5c473\" (UID: \"e9223358-7b05-4518-a90f-e12bbaf5c473\") " Nov 02 15:43:38.651184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.651103 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62a8958-cbe4-431c-aaa6-e2644e34007f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62a8958-cbe4-431c-aaa6-e2644e34007f-plugins-config-volume\") pod \"f62a8958-cbe4-431c-aaa6-e2644e34007f\" (UID: \"f62a8958-cbe4-431c-aaa6-e2644e34007f\") " Nov 02 15:43:38.651184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.651150 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62a8958-cbe4-431c-aaa6-e2644e34007f-kube-api-access-qsmpw\" (UniqueName: \"kubernetes.io/projected/f62a8958-cbe4-431c-aaa6-e2644e34007f-kube-api-access-qsmpw\") pod \"f62a8958-cbe4-431c-aaa6-e2644e34007f\" (UID: \"f62a8958-cbe4-431c-aaa6-e2644e34007f\") " Nov 02 15:43:38.651391 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.651314 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9223358-7b05-4518-a90f-e12bbaf5c473-plugins-config-volume" pod "e9223358-7b05-4518-a90f-e12bbaf5c473" (UID: "e9223358-7b05-4518-a90f-e12bbaf5c473"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:43:38.651604 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.651575 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62a8958-cbe4-431c-aaa6-e2644e34007f-plugins-config-volume" pod "f62a8958-cbe4-431c-aaa6-e2644e34007f" (UID: "f62a8958-cbe4-431c-aaa6-e2644e34007f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:43:38.651825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.651799 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f62a8958-cbe4-431c-aaa6-e2644e34007f-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:38.651947 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.651830 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9223358-7b05-4518-a90f-e12bbaf5c473-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:38.653427 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.653400 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62a8958-cbe4-431c-aaa6-e2644e34007f-kube-api-access-qsmpw" pod "f62a8958-cbe4-431c-aaa6-e2644e34007f" (UID: "f62a8958-cbe4-431c-aaa6-e2644e34007f"). InnerVolumeSpecName "kube-api-access-qsmpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:38.653510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.653407 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9223358-7b05-4518-a90f-e12bbaf5c473-kube-api-access-v9g6m" pod "e9223358-7b05-4518-a90f-e12bbaf5c473" (UID: "e9223358-7b05-4518-a90f-e12bbaf5c473"). InnerVolumeSpecName "kube-api-access-v9g6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:38.753091 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.752971 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsmpw\" (UniqueName: \"kubernetes.io/projected/f62a8958-cbe4-431c-aaa6-e2644e34007f-kube-api-access-qsmpw\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:38.753091 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:38.753041 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9g6m\" (UniqueName: \"kubernetes.io/projected/e9223358-7b05-4518-a90f-e12bbaf5c473-kube-api-access-v9g6m\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:39.216988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.216957 5476 generic.go:356] "Generic (PLEG): container finished" podID="e9223358-7b05-4518-a90f-e12bbaf5c473" containerID="ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746" exitCode=0 Nov 02 15:43:39.217424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.217026 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" Nov 02 15:43:39.217424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.217045 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" event={"ID":"e9223358-7b05-4518-a90f-e12bbaf5c473","Type":"ContainerDied","Data":"ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746"} Nov 02 15:43:39.217424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.217106 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb" event={"ID":"e9223358-7b05-4518-a90f-e12bbaf5c473","Type":"ContainerDied","Data":"fb419c9f4b1f4b88a435c1b6d09eec13dbd6c4f5ff5413502ffedd4750452fe1"} Nov 02 15:43:39.217424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.217138 5476 scope.go:122] "RemoveContainer" containerID="ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746" Nov 02 15:43:39.218760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.218737 5476 generic.go:356] "Generic (PLEG): container finished" podID="f62a8958-cbe4-431c-aaa6-e2644e34007f" containerID="8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d" exitCode=0 Nov 02 15:43:39.218862 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.218797 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" Nov 02 15:43:39.218862 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.218826 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" event={"ID":"f62a8958-cbe4-431c-aaa6-e2644e34007f","Type":"ContainerDied","Data":"8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d"} Nov 02 15:43:39.218952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.218889 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq" event={"ID":"f62a8958-cbe4-431c-aaa6-e2644e34007f","Type":"ContainerDied","Data":"dba3bb370cfc68d8d4e32624577d4f028157d333c55b94a3e3129742dc56fa80"} Nov 02 15:43:39.230831 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.230816 5476 scope.go:122] "RemoveContainer" containerID="ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746" Nov 02 15:43:39.231241 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:43:39.231126 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746\": container with ID starting with ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746 not found: ID does not exist" containerID="ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746" Nov 02 15:43:39.231241 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.231151 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746"} err="failed to get container status \"ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746\": rpc error: code = NotFound desc = could not find container \"ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746\": container with ID starting with ecee74fd45106a4d897136fca93e04e00ac947105c80358ec121af4a1991c746 not found: ID does not exist" Nov 02 15:43:39.231241 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.231173 5476 scope.go:122] "RemoveContainer" containerID="8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d" Nov 02 15:43:39.241504 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.241486 5476 scope.go:122] "RemoveContainer" containerID="8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d" Nov 02 15:43:39.241756 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:43:39.241732 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d\": container with ID starting with 8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d not found: ID does not exist" containerID="8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d" Nov 02 15:43:39.241845 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.241761 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d"} err="failed to get container status \"8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d\": rpc error: code = NotFound desc = could not find container \"8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d\": container with ID starting with 8eace9ceeedc72128363e6954d4e08be022ae84b308337667f946528351ea50d not found: ID does not exist" Nov 02 15:43:39.374378 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.374303 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq"] Nov 02 15:43:39.423119 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.423083 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ca2cf-5bcd876f7f-nj8nq"] Nov 02 15:43:39.483752 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.483667 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb"] Nov 02 15:43:39.486459 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.486430 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ca2cf-6fd74c5fc5-wbbhb"] Nov 02 15:43:39.924277 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.924242 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9223358-7b05-4518-a90f-e12bbaf5c473" path="/var/lib/kubelet/pods/e9223358-7b05-4518-a90f-e12bbaf5c473/volumes" Nov 02 15:43:39.924676 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:39.924662 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62a8958-cbe4-431c-aaa6-e2644e34007f" path="/var/lib/kubelet/pods/f62a8958-cbe4-431c-aaa6-e2644e34007f/volumes" Nov 02 15:43:50.147651 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:43:50.147616 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:43:54.141635 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:43:54.141597 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:44:20.146472 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:44:20.146437 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:44:25.494846 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.494809 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz"] Nov 02 15:44:25.506915 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.506862 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" podUID="04768429-0d60-4444-a82c-e6642e2445d6" containerName="discovery" containerID="cri-o://c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353" gracePeriod=30 Nov 02 15:44:25.789956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.789928 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:44:25.850377 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.850349 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04768429-0d60-4444-a82c-e6642e2445d6-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04768429-0d60-4444-a82c-e6642e2445d6-local-certs\") pod \"04768429-0d60-4444-a82c-e6642e2445d6\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " Nov 02 15:44:25.852383 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.852351 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04768429-0d60-4444-a82c-e6642e2445d6-local-certs" pod "04768429-0d60-4444-a82c-e6642e2445d6" (UID: "04768429-0d60-4444-a82c-e6642e2445d6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:44:25.951343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.951311 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-kubeconfig\") pod \"04768429-0d60-4444-a82c-e6642e2445d6\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " Nov 02 15:44:25.951511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.951444 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-cacerts\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-cacerts\") pod \"04768429-0d60-4444-a82c-e6642e2445d6\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " Nov 02 15:44:25.951556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.951507 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-istio-token\" (UniqueName: \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-istio-token\") pod \"04768429-0d60-4444-a82c-e6642e2445d6\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " Nov 02 15:44:25.951610 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.951565 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-dns-cert\") pod \"04768429-0d60-4444-a82c-e6642e2445d6\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " Nov 02 15:44:25.951678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.951642 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-kube-api-access-dv2xs\" (UniqueName: \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-kube-api-access-dv2xs\") pod \"04768429-0d60-4444-a82c-e6642e2445d6\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " Nov 02 15:44:25.951826 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.951795 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-ca-configmap\") pod \"04768429-0d60-4444-a82c-e6642e2445d6\" (UID: \"04768429-0d60-4444-a82c-e6642e2445d6\") " Nov 02 15:44:25.952240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.952188 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-ca-configmap" pod "04768429-0d60-4444-a82c-e6642e2445d6" (UID: "04768429-0d60-4444-a82c-e6642e2445d6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:44:25.953184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.953155 5476 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-ca-configmap\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:25.953308 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.953189 5476 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04768429-0d60-4444-a82c-e6642e2445d6-local-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:25.954145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.954058 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-dns-cert" pod "04768429-0d60-4444-a82c-e6642e2445d6" (UID: "04768429-0d60-4444-a82c-e6642e2445d6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:44:25.954356 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.954330 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-cacerts" pod "04768429-0d60-4444-a82c-e6642e2445d6" (UID: "04768429-0d60-4444-a82c-e6642e2445d6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:44:25.954472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.954356 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-kubeconfig" pod "04768429-0d60-4444-a82c-e6642e2445d6" (UID: "04768429-0d60-4444-a82c-e6642e2445d6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:44:25.954472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.954373 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-istio-token" pod "04768429-0d60-4444-a82c-e6642e2445d6" (UID: "04768429-0d60-4444-a82c-e6642e2445d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:44:25.954472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:25.954426 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-kube-api-access-dv2xs" pod "04768429-0d60-4444-a82c-e6642e2445d6" (UID: "04768429-0d60-4444-a82c-e6642e2445d6"). InnerVolumeSpecName "kube-api-access-dv2xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:44:26.055370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.055274 5476 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-csr-dns-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:26.055370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.055310 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv2xs\" (UniqueName: \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-kube-api-access-dv2xs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:26.055370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.055329 5476 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-istio-kubeconfig\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:26.055370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.055344 5476 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04768429-0d60-4444-a82c-e6642e2445d6-cacerts\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:26.055370 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.055359 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04768429-0d60-4444-a82c-e6642e2445d6-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:26.514419 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.514379 5476 generic.go:356] "Generic (PLEG): container finished" podID="04768429-0d60-4444-a82c-e6642e2445d6" containerID="c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353" exitCode=0 Nov 02 15:44:26.514845 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.514429 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" Nov 02 15:44:26.514845 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.514455 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" event={"ID":"04768429-0d60-4444-a82c-e6642e2445d6","Type":"ContainerDied","Data":"c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353"} Nov 02 15:44:26.514845 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.514505 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz" event={"ID":"04768429-0d60-4444-a82c-e6642e2445d6","Type":"ContainerDied","Data":"9a3a8e55fc1ae4b4abc51057f0211a1ae1ef456fae677984d6e725a26b808aed"} Nov 02 15:44:26.514845 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.514524 5476 scope.go:122] "RemoveContainer" containerID="c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353" Nov 02 15:44:26.525519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.525499 5476 scope.go:122] "RemoveContainer" containerID="c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353" Nov 02 15:44:26.525811 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:44:26.525792 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353\": container with ID starting with c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353 not found: ID does not exist" containerID="c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353" Nov 02 15:44:26.525881 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.525821 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353"} err="failed to get container status \"c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353\": rpc error: code = NotFound desc = could not find container \"c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353\": container with ID starting with c4be8ce8d3a2a5801f65be26fac7d2e4a1f44d7f0d257b44807f71d8f6c05353 not found: ID does not exist" Nov 02 15:44:26.676387 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.676346 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz"] Nov 02 15:44:26.723473 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:26.723442 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-z7hdz"] Nov 02 15:44:27.922631 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:27.922598 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04768429-0d60-4444-a82c-e6642e2445d6" path="/var/lib/kubelet/pods/04768429-0d60-4444-a82c-e6642e2445d6/volumes" Nov 02 15:44:41.432362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:41.432323 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:44:41.780652 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:41.780573 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg"] Nov 02 15:44:41.780817 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:41.780787 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" podUID="679ebea9-1ba7-4eb1-a9f2-784b99f805f4" containerName="maas-api" containerID="cri-o://90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be" gracePeriod=30 Nov 02 15:44:42.069802 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.069774 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:44:42.143260 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.143223 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-kube-api-access-h5r92\" (UniqueName: \"kubernetes.io/projected/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-kube-api-access-h5r92\") pod \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " Nov 02 15:44:42.143439 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.143377 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-maas-api-tls\") pod \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " Nov 02 15:44:42.143439 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.143407 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-openshift-service-ca\") pod \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " Nov 02 15:44:42.143560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.143519 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-metrics-certs\") pod \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\" (UID: \"679ebea9-1ba7-4eb1-a9f2-784b99f805f4\") " Nov 02 15:44:42.143875 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.143840 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-openshift-service-ca" pod "679ebea9-1ba7-4eb1-a9f2-784b99f805f4" (UID: "679ebea9-1ba7-4eb1-a9f2-784b99f805f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:44:42.144023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.143979 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:42.145465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.145432 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-metrics-certs" pod "679ebea9-1ba7-4eb1-a9f2-784b99f805f4" (UID: "679ebea9-1ba7-4eb1-a9f2-784b99f805f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:44:42.145558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.145491 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-maas-api-tls" pod "679ebea9-1ba7-4eb1-a9f2-784b99f805f4" (UID: "679ebea9-1ba7-4eb1-a9f2-784b99f805f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:44:42.145558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.145494 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-kube-api-access-h5r92" pod "679ebea9-1ba7-4eb1-a9f2-784b99f805f4" (UID: "679ebea9-1ba7-4eb1-a9f2-784b99f805f4"). InnerVolumeSpecName "kube-api-access-h5r92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:44:42.245297 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.245260 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5r92\" (UniqueName: \"kubernetes.io/projected/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-kube-api-access-h5r92\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:42.245297 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.245288 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:42.245297 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.245298 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679ebea9-1ba7-4eb1-a9f2-784b99f805f4-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:42.620184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.620152 5476 generic.go:356] "Generic (PLEG): container finished" podID="679ebea9-1ba7-4eb1-a9f2-784b99f805f4" containerID="90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be" exitCode=0 Nov 02 15:44:42.620663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.620200 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" Nov 02 15:44:42.620663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.620240 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" event={"ID":"679ebea9-1ba7-4eb1-a9f2-784b99f805f4","Type":"ContainerDied","Data":"90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be"} Nov 02 15:44:42.620663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.620302 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg" event={"ID":"679ebea9-1ba7-4eb1-a9f2-784b99f805f4","Type":"ContainerDied","Data":"4777483ec2f1b7c9849941f3310e4ce445a5b1363614a9c5dac8d476b66596da"} Nov 02 15:44:42.620663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.620334 5476 scope.go:122] "RemoveContainer" containerID="90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be" Nov 02 15:44:42.630804 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.630788 5476 scope.go:122] "RemoveContainer" containerID="90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be" Nov 02 15:44:42.631131 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:44:42.631101 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be\": container with ID starting with 90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be not found: ID does not exist" containerID="90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be" Nov 02 15:44:42.631243 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.631140 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be"} err="failed to get container status \"90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be\": rpc error: code = NotFound desc = could not find container \"90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be\": container with ID starting with 90d0cf14da517369bc61a686eb62cda65bc6a40b07df0ce83b4051762612c3be not found: ID does not exist" Nov 02 15:44:42.807796 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.807764 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg"] Nov 02 15:44:42.816188 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:42.816156 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c35ee-99d6c6f68-rzlvg"] Nov 02 15:44:43.922556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:43.922524 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679ebea9-1ba7-4eb1-a9f2-784b99f805f4" path="/var/lib/kubelet/pods/679ebea9-1ba7-4eb1-a9f2-784b99f805f4/volumes" Nov 02 15:44:50.147636 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:44:50.147596 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:44:53.067980 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.067944 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29"] Nov 02 15:44:53.068502 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068486 5476 cpu_manager.go:414] "containerMap: removing container" podUID="679ebea9-1ba7-4eb1-a9f2-784b99f805f4" containerName="maas-api" Nov 02 15:44:53.068548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068503 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679ebea9-1ba7-4eb1-a9f2-784b99f805f4" containerName="maas-api" Nov 02 15:44:53.068548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068521 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerName="istio-proxy" Nov 02 15:44:53.068548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068528 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerName="istio-proxy" Nov 02 15:44:53.068548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068541 5476 cpu_manager.go:414] "containerMap: removing container" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" Nov 02 15:44:53.068548 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068546 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068556 5476 cpu_manager.go:414] "containerMap: removing container" podUID="f62a8958-cbe4-431c-aaa6-e2644e34007f" containerName="payload-pre-processing" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068561 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62a8958-cbe4-431c-aaa6-e2644e34007f" containerName="payload-pre-processing" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068579 5476 cpu_manager.go:414] "containerMap: removing container" podUID="9d123955-d5a6-4b5c-9f09-62fe07a5ddb5" containerName="payload-pre-processing" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068584 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d123955-d5a6-4b5c-9f09-62fe07a5ddb5" containerName="payload-pre-processing" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068594 5476 cpu_manager.go:414] "containerMap: removing container" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068599 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068609 5476 cpu_manager.go:414] "containerMap: removing container" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068616 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068627 5476 cpu_manager.go:414] "containerMap: removing container" podUID="04768429-0d60-4444-a82c-e6642e2445d6" containerName="discovery" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068632 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04768429-0d60-4444-a82c-e6642e2445d6" containerName="discovery" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068643 5476 cpu_manager.go:414] "containerMap: removing container" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068648 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068657 5476 cpu_manager.go:414] "containerMap: removing container" podUID="e9223358-7b05-4518-a90f-e12bbaf5c473" containerName="payload-processing" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068661 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9223358-7b05-4518-a90f-e12bbaf5c473" containerName="payload-processing" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068667 5476 cpu_manager.go:414] "containerMap: removing container" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" Nov 02 15:44:53.068706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068671 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" Nov 02 15:44:53.069214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068751 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" Nov 02 15:44:53.069214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068759 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9223358-7b05-4518-a90f-e12bbaf5c473" containerName="payload-processing" Nov 02 15:44:53.069214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068767 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="037fbba8-633a-4c23-bd5e-d138c3c85732" containerName="istio-proxy" Nov 02 15:44:53.069214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068775 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="679ebea9-1ba7-4eb1-a9f2-784b99f805f4" containerName="maas-api" Nov 02 15:44:53.069214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068786 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c634c862-ab4e-4e94-b08a-aa60620afc31" containerName="istio-proxy" Nov 02 15:44:53.069214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068795 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="50036e4d-3d5c-4bb4-8ea8-98193d52cb16" containerName="main" Nov 02 15:44:53.069214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068802 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62a8958-cbe4-431c-aaa6-e2644e34007f" containerName="payload-pre-processing" Nov 02 15:44:53.069214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068809 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d123955-d5a6-4b5c-9f09-62fe07a5ddb5" containerName="payload-pre-processing" Nov 02 15:44:53.069214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068816 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" Nov 02 15:44:53.069214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068822 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a006dc5-eb85-4bf5-8e94-a8b5b3fe4294" containerName="istio-proxy" Nov 02 15:44:53.069214 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.068827 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="04768429-0d60-4444-a82c-e6642e2445d6" containerName="discovery" Nov 02 15:44:53.071973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.071957 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:53.131508 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.131475 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29"] Nov 02 15:44:53.138726 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.138698 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:44:53.174602 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.174568 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:44:53.179200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.179175 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3b351-52h29\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:53.179327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.179264 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w257v\" (UniqueName: \"kubernetes.io/projected/53424310-bc1b-4988-b546-00a55039a9ec-kube-api-access-w257v\") pod \"maas-api-revoke-keys-e2e-api-b-e3b351-52h29\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:53.179393 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.179377 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3b351-52h29\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:53.280412 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.280367 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3b351-52h29\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:53.280626 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.280499 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3b351-52h29\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:53.280626 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.280576 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w257v\" (UniqueName: \"kubernetes.io/projected/53424310-bc1b-4988-b546-00a55039a9ec-kube-api-access-w257v\") pod \"maas-api-revoke-keys-e2e-api-b-e3b351-52h29\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:53.281184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.281156 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3b351-52h29\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:53.282713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.282695 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3b351-52h29\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:53.342412 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.342346 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w257v\" (UniqueName: \"kubernetes.io/projected/53424310-bc1b-4988-b546-00a55039a9ec-kube-api-access-w257v\") pod \"maas-api-revoke-keys-e2e-api-b-e3b351-52h29\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:53.385662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.385626 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:53.763727 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:53.763691 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29"] Nov 02 15:44:53.769574 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:44:53.769543 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53424310_bc1b_4988_b546_00a55039a9ec.slice/crio-610f7195a5920a7d089718519cb57dc4fcaf174a9226eec75535f60744bfb248 WatchSource:0}: Error finding container 610f7195a5920a7d089718519cb57dc4fcaf174a9226eec75535f60744bfb248: Status 404 returned error can't find the container with id 610f7195a5920a7d089718519cb57dc4fcaf174a9226eec75535f60744bfb248 Nov 02 15:44:54.702979 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:54.702926 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" event={"ID":"53424310-bc1b-4988-b546-00a55039a9ec","Type":"ContainerStarted","Data":"4e28f2f6585f33331c7c03ea0286176c3e4a3daeadd7d39d5ba67e4641ae5d4b"} Nov 02 15:44:54.703484 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:54.703017 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" event={"ID":"53424310-bc1b-4988-b546-00a55039a9ec","Type":"ContainerStarted","Data":"610f7195a5920a7d089718519cb57dc4fcaf174a9226eec75535f60744bfb248"} Nov 02 15:44:54.822745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:54.822691 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" podStartSLOduration=2.8226715220000003 podStartE2EDuration="2.822671522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:54.821778127 +0000 UTC m=+2797.024656186" watchObservedRunningTime="2026-11-02 15:44:54.822671522 +0000 UTC m=+2797.025549582" Nov 02 15:44:55.710959 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:55.710923 5476 generic.go:356] "Generic (PLEG): container finished" podID="53424310-bc1b-4988-b546-00a55039a9ec" containerID="4e28f2f6585f33331c7c03ea0286176c3e4a3daeadd7d39d5ba67e4641ae5d4b" exitCode=0 Nov 02 15:44:55.711387 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:55.710987 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" event={"ID":"53424310-bc1b-4988-b546-00a55039a9ec","Type":"ContainerDied","Data":"4e28f2f6585f33331c7c03ea0286176c3e4a3daeadd7d39d5ba67e4641ae5d4b"} Nov 02 15:44:56.855171 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:56.855143 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:57.030238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.030128 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-openshift-ca\") pod \"53424310-bc1b-4988-b546-00a55039a9ec\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " Nov 02 15:44:57.030382 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.030281 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53424310-bc1b-4988-b546-00a55039a9ec-kube-api-access-w257v\" (UniqueName: \"kubernetes.io/projected/53424310-bc1b-4988-b546-00a55039a9ec-kube-api-access-w257v\") pod \"53424310-bc1b-4988-b546-00a55039a9ec\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " Nov 02 15:44:57.030382 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.030348 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-xks-ca\") pod \"53424310-bc1b-4988-b546-00a55039a9ec\" (UID: \"53424310-bc1b-4988-b546-00a55039a9ec\") " Nov 02 15:44:57.030523 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.030501 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-openshift-ca" pod "53424310-bc1b-4988-b546-00a55039a9ec" (UID: "53424310-bc1b-4988-b546-00a55039a9ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:44:57.030868 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.030846 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.032519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.032488 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53424310-bc1b-4988-b546-00a55039a9ec-kube-api-access-w257v" pod "53424310-bc1b-4988-b546-00a55039a9ec" (UID: "53424310-bc1b-4988-b546-00a55039a9ec"). InnerVolumeSpecName "kube-api-access-w257v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:44:57.032519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.032488 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-xks-ca" pod "53424310-bc1b-4988-b546-00a55039a9ec" (UID: "53424310-bc1b-4988-b546-00a55039a9ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:44:57.132322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.132289 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w257v\" (UniqueName: \"kubernetes.io/projected/53424310-bc1b-4988-b546-00a55039a9ec-kube-api-access-w257v\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.132322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.132317 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53424310-bc1b-4988-b546-00a55039a9ec-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:44:57.727531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.727498 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" Nov 02 15:44:57.727531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.727512 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29" event={"ID":"53424310-bc1b-4988-b546-00a55039a9ec","Type":"ContainerDied","Data":"610f7195a5920a7d089718519cb57dc4fcaf174a9226eec75535f60744bfb248"} Nov 02 15:44:57.727752 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:44:57.727560 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="610f7195a5920a7d089718519cb57dc4fcaf174a9226eec75535f60744bfb248" Nov 02 15:45:00.219671 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.219637 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj"] Nov 02 15:45:00.220256 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.220238 5476 cpu_manager.go:414] "containerMap: removing container" podUID="53424310-bc1b-4988-b546-00a55039a9ec" containerName="revoke-keys" Nov 02 15:45:00.220256 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.220257 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53424310-bc1b-4988-b546-00a55039a9ec" containerName="revoke-keys" Nov 02 15:45:00.220365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.220355 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="53424310-bc1b-4988-b546-00a55039a9ec" containerName="revoke-keys" Nov 02 15:45:00.223693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.223677 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk"] Nov 02 15:45:00.223800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.223783 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" Nov 02 15:45:00.227400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.227376 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" Nov 02 15:45:00.248944 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.248917 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9"] Nov 02 15:45:00.253539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.253518 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" Nov 02 15:45:00.261848 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.261823 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2"] Nov 02 15:45:00.265959 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.265933 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txwvr\" (UniqueName: \"kubernetes.io/projected/3e5c73d7-b426-48b3-bd36-3f628e431f85-kube-api-access-txwvr\") pod \"maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk\" (UID: \"3e5c73d7-b426-48b3-bd36-3f628e431f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" Nov 02 15:45:00.266095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.265969 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8s8v\" (UniqueName: \"kubernetes.io/projected/e884fe81-b9b3-43b9-b873-61603318b613-kube-api-access-b8s8v\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj\" (UID: \"e884fe81-b9b3-43b9-b873-61603318b613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" Nov 02 15:45:00.266203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.266120 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6802668-71af-47b7-8cc8-b2c02d9961de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9\" (UID: \"d6802668-71af-47b7-8cc8-b2c02d9961de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" Nov 02 15:45:00.266203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.266164 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swn6b\" (UniqueName: \"kubernetes.io/projected/d6802668-71af-47b7-8cc8-b2c02d9961de-kube-api-access-swn6b\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9\" (UID: \"d6802668-71af-47b7-8cc8-b2c02d9961de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" Nov 02 15:45:00.266321 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.266257 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5c73d7-b426-48b3-bd36-3f628e431f85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk\" (UID: \"3e5c73d7-b426-48b3-bd36-3f628e431f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" Nov 02 15:45:00.266380 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.266343 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e884fe81-b9b3-43b9-b873-61603318b613-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj\" (UID: \"e884fe81-b9b3-43b9-b873-61603318b613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" Nov 02 15:45:00.269542 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.269524 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg"] Nov 02 15:45:00.269645 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.269630 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" Nov 02 15:45:00.283362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.283332 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" Nov 02 15:45:00.289242 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.289214 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt"] Nov 02 15:45:00.293719 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.293691 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" Nov 02 15:45:00.297076 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.297051 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:45:00.301631 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.301610 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj"] Nov 02 15:45:00.314493 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.314471 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x"] Nov 02 15:45:00.318306 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.318289 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" Nov 02 15:45:00.356918 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.356889 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk"] Nov 02 15:45:00.360874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.360846 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9"] Nov 02 15:45:00.366914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.366888 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txwvr\" (UniqueName: \"kubernetes.io/projected/3e5c73d7-b426-48b3-bd36-3f628e431f85-kube-api-access-txwvr\") pod \"maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk\" (UID: \"3e5c73d7-b426-48b3-bd36-3f628e431f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" Nov 02 15:45:00.367089 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.366919 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8s8v\" (UniqueName: \"kubernetes.io/projected/e884fe81-b9b3-43b9-b873-61603318b613-kube-api-access-b8s8v\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj\" (UID: \"e884fe81-b9b3-43b9-b873-61603318b613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" Nov 02 15:45:00.367089 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.366945 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z5nj\" (UniqueName: \"kubernetes.io/projected/51a8032f-8a4d-412e-a04f-2c479a8a05df-kube-api-access-2z5nj\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x\" (UID: \"51a8032f-8a4d-412e-a04f-2c479a8a05df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" Nov 02 15:45:00.367089 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.366962 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsr27\" (UniqueName: \"kubernetes.io/projected/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-kube-api-access-jsr27\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg\" (UID: \"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" Nov 02 15:45:00.367089 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367072 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6802668-71af-47b7-8cc8-b2c02d9961de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9\" (UID: \"d6802668-71af-47b7-8cc8-b2c02d9961de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" Nov 02 15:45:00.367399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367155 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swn6b\" (UniqueName: \"kubernetes.io/projected/d6802668-71af-47b7-8cc8-b2c02d9961de-kube-api-access-swn6b\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9\" (UID: \"d6802668-71af-47b7-8cc8-b2c02d9961de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" Nov 02 15:45:00.367399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367216 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51a8032f-8a4d-412e-a04f-2c479a8a05df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x\" (UID: \"51a8032f-8a4d-412e-a04f-2c479a8a05df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" Nov 02 15:45:00.367399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367277 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5c73d7-b426-48b3-bd36-3f628e431f85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk\" (UID: \"3e5c73d7-b426-48b3-bd36-3f628e431f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" Nov 02 15:45:00.367399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367349 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e884fe81-b9b3-43b9-b873-61603318b613-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj\" (UID: \"e884fe81-b9b3-43b9-b873-61603318b613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" Nov 02 15:45:00.367399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367375 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff17841-24b5-4837-9a6c-104783665b81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt\" (UID: \"3ff17841-24b5-4837-9a6c-104783665b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" Nov 02 15:45:00.367683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367400 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prq2k\" (UniqueName: \"kubernetes.io/projected/3ff17841-24b5-4837-9a6c-104783665b81-kube-api-access-prq2k\") pod \"maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt\" (UID: \"3ff17841-24b5-4837-9a6c-104783665b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" Nov 02 15:45:00.367683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367491 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b16007-2c3b-4e4c-b08b-4121b3278a65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2\" (UID: \"79b16007-2c3b-4e4c-b08b-4121b3278a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" Nov 02 15:45:00.367683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367574 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg\" (UID: \"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" Nov 02 15:45:00.367683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367627 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g22q\" (UniqueName: \"kubernetes.io/projected/79b16007-2c3b-4e4c-b08b-4121b3278a65-kube-api-access-2g22q\") pod \"maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2\" (UID: \"79b16007-2c3b-4e4c-b08b-4121b3278a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" Nov 02 15:45:00.367899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367835 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6802668-71af-47b7-8cc8-b2c02d9961de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9\" (UID: \"d6802668-71af-47b7-8cc8-b2c02d9961de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" Nov 02 15:45:00.367962 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.367931 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5c73d7-b426-48b3-bd36-3f628e431f85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk\" (UID: \"3e5c73d7-b426-48b3-bd36-3f628e431f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" Nov 02 15:45:00.368074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.368054 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e884fe81-b9b3-43b9-b873-61603318b613-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj\" (UID: \"e884fe81-b9b3-43b9-b873-61603318b613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" Nov 02 15:45:00.374824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.374791 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2"] Nov 02 15:45:00.410709 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.410674 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x"] Nov 02 15:45:00.413525 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.413499 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg"] Nov 02 15:45:00.428601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.428577 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt"] Nov 02 15:45:00.432653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.432627 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8s8v\" (UniqueName: \"kubernetes.io/projected/e884fe81-b9b3-43b9-b873-61603318b613-kube-api-access-b8s8v\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj\" (UID: \"e884fe81-b9b3-43b9-b873-61603318b613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" Nov 02 15:45:00.432796 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.432686 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txwvr\" (UniqueName: \"kubernetes.io/projected/3e5c73d7-b426-48b3-bd36-3f628e431f85-kube-api-access-txwvr\") pod \"maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk\" (UID: \"3e5c73d7-b426-48b3-bd36-3f628e431f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" Nov 02 15:45:00.433572 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.433548 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swn6b\" (UniqueName: \"kubernetes.io/projected/d6802668-71af-47b7-8cc8-b2c02d9961de-kube-api-access-swn6b\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9\" (UID: \"d6802668-71af-47b7-8cc8-b2c02d9961de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" Nov 02 15:45:00.468459 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.468417 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prq2k\" (UniqueName: \"kubernetes.io/projected/3ff17841-24b5-4837-9a6c-104783665b81-kube-api-access-prq2k\") pod \"maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt\" (UID: \"3ff17841-24b5-4837-9a6c-104783665b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" Nov 02 15:45:00.468613 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.468483 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b16007-2c3b-4e4c-b08b-4121b3278a65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2\" (UID: \"79b16007-2c3b-4e4c-b08b-4121b3278a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" Nov 02 15:45:00.468613 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.468520 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg\" (UID: \"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" Nov 02 15:45:00.468613 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.468561 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g22q\" (UniqueName: \"kubernetes.io/projected/79b16007-2c3b-4e4c-b08b-4121b3278a65-kube-api-access-2g22q\") pod \"maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2\" (UID: \"79b16007-2c3b-4e4c-b08b-4121b3278a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" Nov 02 15:45:00.468613 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.468591 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z5nj\" (UniqueName: \"kubernetes.io/projected/51a8032f-8a4d-412e-a04f-2c479a8a05df-kube-api-access-2z5nj\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x\" (UID: \"51a8032f-8a4d-412e-a04f-2c479a8a05df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" Nov 02 15:45:00.468613 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.468612 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsr27\" (UniqueName: \"kubernetes.io/projected/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-kube-api-access-jsr27\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg\" (UID: \"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" Nov 02 15:45:00.469023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.468691 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51a8032f-8a4d-412e-a04f-2c479a8a05df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x\" (UID: \"51a8032f-8a4d-412e-a04f-2c479a8a05df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" Nov 02 15:45:00.469023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.468755 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff17841-24b5-4837-9a6c-104783665b81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt\" (UID: \"3ff17841-24b5-4837-9a6c-104783665b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" Nov 02 15:45:00.469346 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.469324 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg\" (UID: \"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" Nov 02 15:45:00.469596 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.469571 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51a8032f-8a4d-412e-a04f-2c479a8a05df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x\" (UID: \"51a8032f-8a4d-412e-a04f-2c479a8a05df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" Nov 02 15:45:00.469848 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.469801 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b16007-2c3b-4e4c-b08b-4121b3278a65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2\" (UID: \"79b16007-2c3b-4e4c-b08b-4121b3278a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" Nov 02 15:45:00.469910 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.469874 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff17841-24b5-4837-9a6c-104783665b81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt\" (UID: \"3ff17841-24b5-4837-9a6c-104783665b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" Nov 02 15:45:00.536513 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.536477 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g22q\" (UniqueName: \"kubernetes.io/projected/79b16007-2c3b-4e4c-b08b-4121b3278a65-kube-api-access-2g22q\") pod \"maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2\" (UID: \"79b16007-2c3b-4e4c-b08b-4121b3278a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" Nov 02 15:45:00.536790 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.536767 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z5nj\" (UniqueName: \"kubernetes.io/projected/51a8032f-8a4d-412e-a04f-2c479a8a05df-kube-api-access-2z5nj\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x\" (UID: \"51a8032f-8a4d-412e-a04f-2c479a8a05df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" Nov 02 15:45:00.536891 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.536873 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prq2k\" (UniqueName: \"kubernetes.io/projected/3ff17841-24b5-4837-9a6c-104783665b81-kube-api-access-prq2k\") pod \"maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt\" (UID: \"3ff17841-24b5-4837-9a6c-104783665b81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" Nov 02 15:45:00.541286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.541268 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" Nov 02 15:45:00.543436 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.543412 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsr27\" (UniqueName: \"kubernetes.io/projected/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-kube-api-access-jsr27\") pod \"maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg\" (UID: \"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" Nov 02 15:45:00.549726 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.549709 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" Nov 02 15:45:00.567929 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.567733 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" Nov 02 15:45:00.598354 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.598188 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" Nov 02 15:45:00.612288 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.607181 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" Nov 02 15:45:00.612288 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.612232 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" Nov 02 15:45:00.637466 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.637302 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" Nov 02 15:45:00.940385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:00.940350 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj"] Nov 02 15:45:00.944809 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:45:00.944761 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode884fe81_b9b3_43b9_b873_61603318b613.slice/crio-e097ee1592f7e5d9513e0cf471703a6ef9fea27df00b703aff4083f785642764 WatchSource:0}: Error finding container e097ee1592f7e5d9513e0cf471703a6ef9fea27df00b703aff4083f785642764: Status 404 returned error can't find the container with id e097ee1592f7e5d9513e0cf471703a6ef9fea27df00b703aff4083f785642764 Nov 02 15:45:01.023806 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.023717 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2"] Nov 02 15:45:01.030373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.026752 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9"] Nov 02 15:45:01.030373 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:45:01.029063 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6802668_71af_47b7_8cc8_b2c02d9961de.slice/crio-f623cedf908f168a89ea3bf9f082d5d1ad8659b43ad8b390ad01fbb4eeee37f3 WatchSource:0}: Error finding container f623cedf908f168a89ea3bf9f082d5d1ad8659b43ad8b390ad01fbb4eeee37f3: Status 404 returned error can't find the container with id f623cedf908f168a89ea3bf9f082d5d1ad8659b43ad8b390ad01fbb4eeee37f3 Nov 02 15:45:01.030493 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:45:01.030396 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b16007_2c3b_4e4c_b08b_4121b3278a65.slice/crio-8bb499fa6be03d5b76c1d60faf704a3aedc7ea74783dc2a3b8378c051b50eadd WatchSource:0}: Error finding container 8bb499fa6be03d5b76c1d60faf704a3aedc7ea74783dc2a3b8378c051b50eadd: Status 404 returned error can't find the container with id 8bb499fa6be03d5b76c1d60faf704a3aedc7ea74783dc2a3b8378c051b50eadd Nov 02 15:45:01.073899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.073858 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg"] Nov 02 15:45:01.077075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.077054 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt"] Nov 02 15:45:01.113779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.113700 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x"] Nov 02 15:45:01.118904 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.118883 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk"] Nov 02 15:45:01.175405 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:45:01.175375 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e92d4a_de8d_4244_912c_9b6d2ddf4b8a.slice/crio-1ce09f7a94dd9878eeb396064e74359401ca233a1ef1c2968857ab04b1153bf3 WatchSource:0}: Error finding container 1ce09f7a94dd9878eeb396064e74359401ca233a1ef1c2968857ab04b1153bf3: Status 404 returned error can't find the container with id 1ce09f7a94dd9878eeb396064e74359401ca233a1ef1c2968857ab04b1153bf3 Nov 02 15:45:01.176945 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:45:01.176919 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff17841_24b5_4837_9a6c_104783665b81.slice/crio-08b9f3cb8ceba711c0e692cac717e236005a2942ef8855833e30e891c78968b6 WatchSource:0}: Error finding container 08b9f3cb8ceba711c0e692cac717e236005a2942ef8855833e30e891c78968b6: Status 404 returned error can't find the container with id 08b9f3cb8ceba711c0e692cac717e236005a2942ef8855833e30e891c78968b6 Nov 02 15:45:01.185355 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:45:01.185331 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5c73d7_b426_48b3_bd36_3f628e431f85.slice/crio-48a24e7d86b47256ce0c4720ca29ad8da48f9c6a78b899e5a9452ce893115bb5 WatchSource:0}: Error finding container 48a24e7d86b47256ce0c4720ca29ad8da48f9c6a78b899e5a9452ce893115bb5: Status 404 returned error can't find the container with id 48a24e7d86b47256ce0c4720ca29ad8da48f9c6a78b899e5a9452ce893115bb5 Nov 02 15:45:01.185579 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:45:01.185556 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a8032f_8a4d_412e_a04f_2c479a8a05df.slice/crio-d10ef12d97ff980211f586307bb610eb241a80a6bec6472339cc17361143e21d WatchSource:0}: Error finding container d10ef12d97ff980211f586307bb610eb241a80a6bec6472339cc17361143e21d: Status 404 returned error can't find the container with id d10ef12d97ff980211f586307bb610eb241a80a6bec6472339cc17361143e21d Nov 02 15:45:01.765939 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.765901 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" event={"ID":"79b16007-2c3b-4e4c-b08b-4121b3278a65","Type":"ContainerStarted","Data":"d5ee5423e96c2d387307cd213247164f72c8904c83e6640bd67fec9c8cab231b"} Nov 02 15:45:01.766434 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.765989 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" event={"ID":"79b16007-2c3b-4e4c-b08b-4121b3278a65","Type":"ContainerStarted","Data":"8bb499fa6be03d5b76c1d60faf704a3aedc7ea74783dc2a3b8378c051b50eadd"} Nov 02 15:45:01.767611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.767581 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" event={"ID":"d6802668-71af-47b7-8cc8-b2c02d9961de","Type":"ContainerStarted","Data":"2e1fe5ac3853577ddd4d52020b0b1e28ed9f054945b8b5e8de155949fef3dcce"} Nov 02 15:45:01.767760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.767636 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" event={"ID":"d6802668-71af-47b7-8cc8-b2c02d9961de","Type":"ContainerStarted","Data":"f623cedf908f168a89ea3bf9f082d5d1ad8659b43ad8b390ad01fbb4eeee37f3"} Nov 02 15:45:01.769189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.769161 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" event={"ID":"3e5c73d7-b426-48b3-bd36-3f628e431f85","Type":"ContainerStarted","Data":"d863bd137ffdf4721661b125178913e692922e2b3d514fb645840b28b6c733db"} Nov 02 15:45:01.769317 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.769210 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" event={"ID":"3e5c73d7-b426-48b3-bd36-3f628e431f85","Type":"ContainerStarted","Data":"48a24e7d86b47256ce0c4720ca29ad8da48f9c6a78b899e5a9452ce893115bb5"} Nov 02 15:45:01.770580 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.770558 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" event={"ID":"3ff17841-24b5-4837-9a6c-104783665b81","Type":"ContainerStarted","Data":"21a2cf46f5c76165e6f9b9badfe951d87f45ab70ce191716a57d34fe8927c805"} Nov 02 15:45:01.770680 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.770600 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" event={"ID":"3ff17841-24b5-4837-9a6c-104783665b81","Type":"ContainerStarted","Data":"08b9f3cb8ceba711c0e692cac717e236005a2942ef8855833e30e891c78968b6"} Nov 02 15:45:01.777145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.777118 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" event={"ID":"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a","Type":"ContainerStarted","Data":"74e910dc7c1e85547daab8237991dce5763aa927463127f33c4850fa7547fda2"} Nov 02 15:45:01.777285 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.777168 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" event={"ID":"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a","Type":"ContainerStarted","Data":"1ce09f7a94dd9878eeb396064e74359401ca233a1ef1c2968857ab04b1153bf3"} Nov 02 15:45:01.778677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.778655 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" event={"ID":"51a8032f-8a4d-412e-a04f-2c479a8a05df","Type":"ContainerStarted","Data":"f03a3d60c91776ea1c2eeacf8ea4a5cabe6cc5e88626d33712ff713234764597"} Nov 02 15:45:01.778797 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.778695 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" event={"ID":"51a8032f-8a4d-412e-a04f-2c479a8a05df","Type":"ContainerStarted","Data":"d10ef12d97ff980211f586307bb610eb241a80a6bec6472339cc17361143e21d"} Nov 02 15:45:01.780215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.780192 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" event={"ID":"e884fe81-b9b3-43b9-b873-61603318b613","Type":"ContainerStarted","Data":"e5a23bc5dbe808e3fdb3a7edbb3e4e7dea1d1cdd0b0b58fd0c509dcd8064829d"} Nov 02 15:45:01.780316 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.780237 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" event={"ID":"e884fe81-b9b3-43b9-b873-61603318b613","Type":"ContainerStarted","Data":"e097ee1592f7e5d9513e0cf471703a6ef9fea27df00b703aff4083f785642764"} Nov 02 15:45:01.884459 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.884377 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" podStartSLOduration=1.8843555379999999 podStartE2EDuration="1.884355538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:01.878211693 +0000 UTC m=+2804.081089753" watchObservedRunningTime="2026-11-02 15:45:01.884355538 +0000 UTC m=+2804.087233599" Nov 02 15:45:01.991529 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:01.991482 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" podStartSLOduration=1.9914679579999999 podStartE2EDuration="1.991467958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:01.988982668 +0000 UTC m=+2804.191860727" watchObservedRunningTime="2026-11-02 15:45:01.991467958 +0000 UTC m=+2804.194346051" Nov 02 15:45:02.105633 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.105578 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" podStartSLOduration=2.105563784 podStartE2EDuration="2.105563784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.104140053 +0000 UTC m=+2804.307018123" watchObservedRunningTime="2026-11-02 15:45:02.105563784 +0000 UTC m=+2804.308441842" Nov 02 15:45:02.222748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.222700 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" podStartSLOduration=2.222685807 podStartE2EDuration="2.222685807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.221825641 +0000 UTC m=+2804.424703710" watchObservedRunningTime="2026-11-02 15:45:02.222685807 +0000 UTC m=+2804.425563865" Nov 02 15:45:02.338077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.338005 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" podStartSLOduration=2.337977025 podStartE2EDuration="2.337977025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.335081768 +0000 UTC m=+2804.537959826" watchObservedRunningTime="2026-11-02 15:45:02.337977025 +0000 UTC m=+2804.540855083" Nov 02 15:45:02.465856 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.465795 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" podStartSLOduration=2.465777649 podStartE2EDuration="2.465777649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.463972492 +0000 UTC m=+2804.666850549" watchObservedRunningTime="2026-11-02 15:45:02.465777649 +0000 UTC m=+2804.668655709" Nov 02 15:45:02.580775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.580720 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" podStartSLOduration=2.580706083 podStartE2EDuration="2.580706083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.579613447 +0000 UTC m=+2804.782491507" watchObservedRunningTime="2026-11-02 15:45:02.580706083 +0000 UTC m=+2804.783584140" Nov 02 15:45:02.789190 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.789107 5476 generic.go:356] "Generic (PLEG): container finished" podID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerID="f03a3d60c91776ea1c2eeacf8ea4a5cabe6cc5e88626d33712ff713234764597" exitCode=60 Nov 02 15:45:02.789633 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.789187 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" event={"ID":"51a8032f-8a4d-412e-a04f-2c479a8a05df","Type":"ContainerDied","Data":"f03a3d60c91776ea1c2eeacf8ea4a5cabe6cc5e88626d33712ff713234764597"} Nov 02 15:45:02.789633 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.789468 5476 scope.go:122] "RemoveContainer" containerID="f03a3d60c91776ea1c2eeacf8ea4a5cabe6cc5e88626d33712ff713234764597" Nov 02 15:45:02.790812 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.790793 5476 generic.go:356] "Generic (PLEG): container finished" podID="e884fe81-b9b3-43b9-b873-61603318b613" containerID="e5a23bc5dbe808e3fdb3a7edbb3e4e7dea1d1cdd0b0b58fd0c509dcd8064829d" exitCode=60 Nov 02 15:45:02.790888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.790868 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" event={"ID":"e884fe81-b9b3-43b9-b873-61603318b613","Type":"ContainerDied","Data":"e5a23bc5dbe808e3fdb3a7edbb3e4e7dea1d1cdd0b0b58fd0c509dcd8064829d"} Nov 02 15:45:02.791126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.791101 5476 scope.go:122] "RemoveContainer" containerID="e5a23bc5dbe808e3fdb3a7edbb3e4e7dea1d1cdd0b0b58fd0c509dcd8064829d" Nov 02 15:45:02.792804 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.792782 5476 generic.go:356] "Generic (PLEG): container finished" podID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerID="d5ee5423e96c2d387307cd213247164f72c8904c83e6640bd67fec9c8cab231b" exitCode=60 Nov 02 15:45:02.792969 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.792888 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" event={"ID":"79b16007-2c3b-4e4c-b08b-4121b3278a65","Type":"ContainerDied","Data":"d5ee5423e96c2d387307cd213247164f72c8904c83e6640bd67fec9c8cab231b"} Nov 02 15:45:02.793117 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.793098 5476 scope.go:122] "RemoveContainer" containerID="d5ee5423e96c2d387307cd213247164f72c8904c83e6640bd67fec9c8cab231b" Nov 02 15:45:02.794811 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.794793 5476 generic.go:356] "Generic (PLEG): container finished" podID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerID="2e1fe5ac3853577ddd4d52020b0b1e28ed9f054945b8b5e8de155949fef3dcce" exitCode=60 Nov 02 15:45:02.794893 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.794841 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" event={"ID":"d6802668-71af-47b7-8cc8-b2c02d9961de","Type":"ContainerDied","Data":"2e1fe5ac3853577ddd4d52020b0b1e28ed9f054945b8b5e8de155949fef3dcce"} Nov 02 15:45:02.795124 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.795104 5476 scope.go:122] "RemoveContainer" containerID="2e1fe5ac3853577ddd4d52020b0b1e28ed9f054945b8b5e8de155949fef3dcce" Nov 02 15:45:02.796474 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.796457 5476 generic.go:356] "Generic (PLEG): container finished" podID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerID="d863bd137ffdf4721661b125178913e692922e2b3d514fb645840b28b6c733db" exitCode=60 Nov 02 15:45:02.796554 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.796506 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" event={"ID":"3e5c73d7-b426-48b3-bd36-3f628e431f85","Type":"ContainerDied","Data":"d863bd137ffdf4721661b125178913e692922e2b3d514fb645840b28b6c733db"} Nov 02 15:45:02.796710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.796697 5476 scope.go:122] "RemoveContainer" containerID="d863bd137ffdf4721661b125178913e692922e2b3d514fb645840b28b6c733db" Nov 02 15:45:02.798120 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.798098 5476 generic.go:356] "Generic (PLEG): container finished" podID="3ff17841-24b5-4837-9a6c-104783665b81" containerID="21a2cf46f5c76165e6f9b9badfe951d87f45ab70ce191716a57d34fe8927c805" exitCode=60 Nov 02 15:45:02.798186 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.798173 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" event={"ID":"3ff17841-24b5-4837-9a6c-104783665b81","Type":"ContainerDied","Data":"21a2cf46f5c76165e6f9b9badfe951d87f45ab70ce191716a57d34fe8927c805"} Nov 02 15:45:02.798344 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.798326 5476 scope.go:122] "RemoveContainer" containerID="21a2cf46f5c76165e6f9b9badfe951d87f45ab70ce191716a57d34fe8927c805" Nov 02 15:45:02.799830 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.799808 5476 generic.go:356] "Generic (PLEG): container finished" podID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerID="74e910dc7c1e85547daab8237991dce5763aa927463127f33c4850fa7547fda2" exitCode=60 Nov 02 15:45:02.799941 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.799884 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" event={"ID":"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a","Type":"ContainerDied","Data":"74e910dc7c1e85547daab8237991dce5763aa927463127f33c4850fa7547fda2"} Nov 02 15:45:02.800197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:02.800180 5476 scope.go:122] "RemoveContainer" containerID="74e910dc7c1e85547daab8237991dce5763aa927463127f33c4850fa7547fda2" Nov 02 15:45:03.809433 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:03.809399 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" event={"ID":"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a","Type":"ContainerStarted","Data":"18529154d68848fce006ff85cb4d78ce4cabe79477aa761e5414e3c159f5a0e7"} Nov 02 15:45:03.811788 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:03.811758 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" event={"ID":"51a8032f-8a4d-412e-a04f-2c479a8a05df","Type":"ContainerStarted","Data":"fd728d2d956d4f5fa8f6a3469a048a53640a275c6fc696ef6e4e8d00ccba05f9"} Nov 02 15:45:03.814087 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:03.814057 5476 generic.go:356] "Generic (PLEG): container finished" podID="e884fe81-b9b3-43b9-b873-61603318b613" containerID="35e8c9b6a752646288344763fe774ca128373ce56b869b45e259bddda3b172a4" exitCode=60 Nov 02 15:45:03.814195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:03.814176 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" event={"ID":"e884fe81-b9b3-43b9-b873-61603318b613","Type":"ContainerDied","Data":"35e8c9b6a752646288344763fe774ca128373ce56b869b45e259bddda3b172a4"} Nov 02 15:45:03.814278 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:03.814235 5476 scope.go:122] "RemoveContainer" containerID="e5a23bc5dbe808e3fdb3a7edbb3e4e7dea1d1cdd0b0b58fd0c509dcd8064829d" Nov 02 15:45:03.814341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:03.814327 5476 scope.go:122] "RemoveContainer" containerID="35e8c9b6a752646288344763fe774ca128373ce56b869b45e259bddda3b172a4" Nov 02 15:45:03.814635 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:03.814615 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj_odh-ai-gateway-infra(e884fe81-b9b3-43b9-b873-61603318b613)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" podUID="e884fe81-b9b3-43b9-b873-61603318b613" Nov 02 15:45:03.816328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:03.816303 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" event={"ID":"79b16007-2c3b-4e4c-b08b-4121b3278a65","Type":"ContainerStarted","Data":"62953b3f759a5ee9d85c89c628fabc680ed946dfa34ac354c8dbaf12df5dcee6"} Nov 02 15:45:03.818542 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:03.818521 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" event={"ID":"d6802668-71af-47b7-8cc8-b2c02d9961de","Type":"ContainerStarted","Data":"08f3efa9d6d8d7accc7b9ebad4a8f75975b62105cd038e69caf0cdef76b9d3b2"} Nov 02 15:45:03.820977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:03.820951 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" event={"ID":"3e5c73d7-b426-48b3-bd36-3f628e431f85","Type":"ContainerStarted","Data":"cae9285ae52f0898ac27bf349cb1069130f622fbabf5535fc3377b5552d1a738"} Nov 02 15:45:03.822774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:03.822754 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" event={"ID":"3ff17841-24b5-4837-9a6c-104783665b81","Type":"ContainerStarted","Data":"512542d40ed84def36cbfe9fc3fc55f49c419f7edbbe45e0db91a799d2a0c071"} Nov 02 15:45:04.831686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.831647 5476 generic.go:356] "Generic (PLEG): container finished" podID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerID="fd728d2d956d4f5fa8f6a3469a048a53640a275c6fc696ef6e4e8d00ccba05f9" exitCode=60 Nov 02 15:45:04.832169 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.831729 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" event={"ID":"51a8032f-8a4d-412e-a04f-2c479a8a05df","Type":"ContainerDied","Data":"fd728d2d956d4f5fa8f6a3469a048a53640a275c6fc696ef6e4e8d00ccba05f9"} Nov 02 15:45:04.832169 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.831835 5476 scope.go:122] "RemoveContainer" containerID="f03a3d60c91776ea1c2eeacf8ea4a5cabe6cc5e88626d33712ff713234764597" Nov 02 15:45:04.832169 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.831960 5476 scope.go:122] "RemoveContainer" containerID="fd728d2d956d4f5fa8f6a3469a048a53640a275c6fc696ef6e4e8d00ccba05f9" Nov 02 15:45:04.832322 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:04.832280 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x_odh-ai-gateway-infra(51a8032f-8a4d-412e-a04f-2c479a8a05df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" Nov 02 15:45:04.833777 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.833754 5476 scope.go:122] "RemoveContainer" containerID="35e8c9b6a752646288344763fe774ca128373ce56b869b45e259bddda3b172a4" Nov 02 15:45:04.833973 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:04.833955 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj_odh-ai-gateway-infra(e884fe81-b9b3-43b9-b873-61603318b613)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" podUID="e884fe81-b9b3-43b9-b873-61603318b613" Nov 02 15:45:04.835407 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.835387 5476 generic.go:356] "Generic (PLEG): container finished" podID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerID="62953b3f759a5ee9d85c89c628fabc680ed946dfa34ac354c8dbaf12df5dcee6" exitCode=60 Nov 02 15:45:04.835522 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.835505 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" event={"ID":"79b16007-2c3b-4e4c-b08b-4121b3278a65","Type":"ContainerDied","Data":"62953b3f759a5ee9d85c89c628fabc680ed946dfa34ac354c8dbaf12df5dcee6"} Nov 02 15:45:04.835770 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.835746 5476 scope.go:122] "RemoveContainer" containerID="62953b3f759a5ee9d85c89c628fabc680ed946dfa34ac354c8dbaf12df5dcee6" Nov 02 15:45:04.836018 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:04.835970 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2_odh-ai-gateway-infra(79b16007-2c3b-4e4c-b08b-4121b3278a65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" Nov 02 15:45:04.837559 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.837536 5476 generic.go:356] "Generic (PLEG): container finished" podID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerID="08f3efa9d6d8d7accc7b9ebad4a8f75975b62105cd038e69caf0cdef76b9d3b2" exitCode=60 Nov 02 15:45:04.837653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.837630 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" event={"ID":"d6802668-71af-47b7-8cc8-b2c02d9961de","Type":"ContainerDied","Data":"08f3efa9d6d8d7accc7b9ebad4a8f75975b62105cd038e69caf0cdef76b9d3b2"} Nov 02 15:45:04.837831 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.837814 5476 scope.go:122] "RemoveContainer" containerID="08f3efa9d6d8d7accc7b9ebad4a8f75975b62105cd038e69caf0cdef76b9d3b2" Nov 02 15:45:04.838145 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:04.838122 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9_odh-ai-gateway-infra(d6802668-71af-47b7-8cc8-b2c02d9961de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" Nov 02 15:45:04.839673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.839652 5476 generic.go:356] "Generic (PLEG): container finished" podID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerID="cae9285ae52f0898ac27bf349cb1069130f622fbabf5535fc3377b5552d1a738" exitCode=60 Nov 02 15:45:04.839775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.839732 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" event={"ID":"3e5c73d7-b426-48b3-bd36-3f628e431f85","Type":"ContainerDied","Data":"cae9285ae52f0898ac27bf349cb1069130f622fbabf5535fc3377b5552d1a738"} Nov 02 15:45:04.839931 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.839912 5476 scope.go:122] "RemoveContainer" containerID="cae9285ae52f0898ac27bf349cb1069130f622fbabf5535fc3377b5552d1a738" Nov 02 15:45:04.840166 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:04.840142 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk_odh-ai-gateway-infra(3e5c73d7-b426-48b3-bd36-3f628e431f85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" Nov 02 15:45:04.841766 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.841742 5476 generic.go:356] "Generic (PLEG): container finished" podID="3ff17841-24b5-4837-9a6c-104783665b81" containerID="512542d40ed84def36cbfe9fc3fc55f49c419f7edbbe45e0db91a799d2a0c071" exitCode=60 Nov 02 15:45:04.841858 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.841817 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" event={"ID":"3ff17841-24b5-4837-9a6c-104783665b81","Type":"ContainerDied","Data":"512542d40ed84def36cbfe9fc3fc55f49c419f7edbbe45e0db91a799d2a0c071"} Nov 02 15:45:04.842040 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.842026 5476 scope.go:122] "RemoveContainer" containerID="512542d40ed84def36cbfe9fc3fc55f49c419f7edbbe45e0db91a799d2a0c071" Nov 02 15:45:04.842255 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:04.842232 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt_odh-ai-gateway-infra(3ff17841-24b5-4837-9a6c-104783665b81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" podUID="3ff17841-24b5-4837-9a6c-104783665b81" Nov 02 15:45:04.843787 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.843766 5476 generic.go:356] "Generic (PLEG): container finished" podID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerID="18529154d68848fce006ff85cb4d78ce4cabe79477aa761e5414e3c159f5a0e7" exitCode=60 Nov 02 15:45:04.843888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.843796 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" event={"ID":"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a","Type":"ContainerDied","Data":"18529154d68848fce006ff85cb4d78ce4cabe79477aa761e5414e3c159f5a0e7"} Nov 02 15:45:04.844077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.844061 5476 scope.go:122] "RemoveContainer" containerID="18529154d68848fce006ff85cb4d78ce4cabe79477aa761e5414e3c159f5a0e7" Nov 02 15:45:04.844300 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:04.844280 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg_odh-ai-gateway-infra(12e92d4a-de8d-4244-912c-9b6d2ddf4b8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" Nov 02 15:45:04.845943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.845926 5476 scope.go:122] "RemoveContainer" containerID="d5ee5423e96c2d387307cd213247164f72c8904c83e6640bd67fec9c8cab231b" Nov 02 15:45:04.858986 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.858966 5476 scope.go:122] "RemoveContainer" containerID="2e1fe5ac3853577ddd4d52020b0b1e28ed9f054945b8b5e8de155949fef3dcce" Nov 02 15:45:04.874163 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.874068 5476 scope.go:122] "RemoveContainer" containerID="d863bd137ffdf4721661b125178913e692922e2b3d514fb645840b28b6c733db" Nov 02 15:45:04.891126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.891049 5476 scope.go:122] "RemoveContainer" containerID="21a2cf46f5c76165e6f9b9badfe951d87f45ab70ce191716a57d34fe8927c805" Nov 02 15:45:04.904502 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:04.904482 5476 scope.go:122] "RemoveContainer" containerID="74e910dc7c1e85547daab8237991dce5763aa927463127f33c4850fa7547fda2" Nov 02 15:45:05.852908 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:05.852880 5476 scope.go:122] "RemoveContainer" containerID="fd728d2d956d4f5fa8f6a3469a048a53640a275c6fc696ef6e4e8d00ccba05f9" Nov 02 15:45:05.853426 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:05.853217 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x_odh-ai-gateway-infra(51a8032f-8a4d-412e-a04f-2c479a8a05df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" Nov 02 15:45:05.854539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:05.854508 5476 scope.go:122] "RemoveContainer" containerID="62953b3f759a5ee9d85c89c628fabc680ed946dfa34ac354c8dbaf12df5dcee6" Nov 02 15:45:05.854838 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:05.854813 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2_odh-ai-gateway-infra(79b16007-2c3b-4e4c-b08b-4121b3278a65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" Nov 02 15:45:05.856181 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:05.856163 5476 scope.go:122] "RemoveContainer" containerID="08f3efa9d6d8d7accc7b9ebad4a8f75975b62105cd038e69caf0cdef76b9d3b2" Nov 02 15:45:05.856427 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:05.856400 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9_odh-ai-gateway-infra(d6802668-71af-47b7-8cc8-b2c02d9961de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" Nov 02 15:45:05.857722 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:05.857702 5476 scope.go:122] "RemoveContainer" containerID="cae9285ae52f0898ac27bf349cb1069130f622fbabf5535fc3377b5552d1a738" Nov 02 15:45:05.857966 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:05.857944 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk_odh-ai-gateway-infra(3e5c73d7-b426-48b3-bd36-3f628e431f85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" Nov 02 15:45:05.859424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:05.859406 5476 scope.go:122] "RemoveContainer" containerID="512542d40ed84def36cbfe9fc3fc55f49c419f7edbbe45e0db91a799d2a0c071" Nov 02 15:45:05.859649 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:05.859617 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt_odh-ai-gateway-infra(3ff17841-24b5-4837-9a6c-104783665b81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" podUID="3ff17841-24b5-4837-9a6c-104783665b81" Nov 02 15:45:05.860990 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:05.860974 5476 scope.go:122] "RemoveContainer" containerID="18529154d68848fce006ff85cb4d78ce4cabe79477aa761e5414e3c159f5a0e7" Nov 02 15:45:05.861183 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:05.861167 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg_odh-ai-gateway-infra(12e92d4a-de8d-4244-912c-9b6d2ddf4b8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" Nov 02 15:45:14.205795 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:14.205757 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:45:14.915252 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:14.915222 5476 scope.go:122] "RemoveContainer" containerID="08f3efa9d6d8d7accc7b9ebad4a8f75975b62105cd038e69caf0cdef76b9d3b2" Nov 02 15:45:15.915624 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:15.915593 5476 scope.go:122] "RemoveContainer" containerID="18529154d68848fce006ff85cb4d78ce4cabe79477aa761e5414e3c159f5a0e7" Nov 02 15:45:15.933696 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:15.933662 5476 generic.go:356] "Generic (PLEG): container finished" podID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerID="8ec0b5ec94c3e33108d9ee05c95673e2a80befec74331875becd99fd8d911f7d" exitCode=60 Nov 02 15:45:15.933806 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:15.933701 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" event={"ID":"d6802668-71af-47b7-8cc8-b2c02d9961de","Type":"ContainerDied","Data":"8ec0b5ec94c3e33108d9ee05c95673e2a80befec74331875becd99fd8d911f7d"} Nov 02 15:45:15.933806 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:15.933767 5476 scope.go:122] "RemoveContainer" containerID="08f3efa9d6d8d7accc7b9ebad4a8f75975b62105cd038e69caf0cdef76b9d3b2" Nov 02 15:45:15.934252 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:15.934231 5476 scope.go:122] "RemoveContainer" containerID="8ec0b5ec94c3e33108d9ee05c95673e2a80befec74331875becd99fd8d911f7d" Nov 02 15:45:15.934592 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:15.934571 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9_odh-ai-gateway-infra(d6802668-71af-47b7-8cc8-b2c02d9961de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" Nov 02 15:45:16.942806 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:16.942724 5476 generic.go:356] "Generic (PLEG): container finished" podID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerID="ae89dc5c99e7ae15b949b011ce33a4221727dfdce3edc27cf443cb88f666c0da" exitCode=60 Nov 02 15:45:16.942806 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:16.942795 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" event={"ID":"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a","Type":"ContainerDied","Data":"ae89dc5c99e7ae15b949b011ce33a4221727dfdce3edc27cf443cb88f666c0da"} Nov 02 15:45:16.943343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:16.942859 5476 scope.go:122] "RemoveContainer" containerID="18529154d68848fce006ff85cb4d78ce4cabe79477aa761e5414e3c159f5a0e7" Nov 02 15:45:16.943343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:16.943294 5476 scope.go:122] "RemoveContainer" containerID="ae89dc5c99e7ae15b949b011ce33a4221727dfdce3edc27cf443cb88f666c0da" Nov 02 15:45:16.943641 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:16.943618 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg_odh-ai-gateway-infra(12e92d4a-de8d-4244-912c-9b6d2ddf4b8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" Nov 02 15:45:17.083396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.083355 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9"] Nov 02 15:45:17.226913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.226888 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" Nov 02 15:45:17.291461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.291429 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6802668-71af-47b7-8cc8-b2c02d9961de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6802668-71af-47b7-8cc8-b2c02d9961de-maas-api-service-ca\") pod \"d6802668-71af-47b7-8cc8-b2c02d9961de\" (UID: \"d6802668-71af-47b7-8cc8-b2c02d9961de\") " Nov 02 15:45:17.291605 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.291495 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6802668-71af-47b7-8cc8-b2c02d9961de-kube-api-access-swn6b\" (UniqueName: \"kubernetes.io/projected/d6802668-71af-47b7-8cc8-b2c02d9961de-kube-api-access-swn6b\") pod \"d6802668-71af-47b7-8cc8-b2c02d9961de\" (UID: \"d6802668-71af-47b7-8cc8-b2c02d9961de\") " Nov 02 15:45:17.291794 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.291774 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6802668-71af-47b7-8cc8-b2c02d9961de-maas-api-service-ca" pod "d6802668-71af-47b7-8cc8-b2c02d9961de" (UID: "d6802668-71af-47b7-8cc8-b2c02d9961de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:17.293546 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.293519 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6802668-71af-47b7-8cc8-b2c02d9961de-kube-api-access-swn6b" pod "d6802668-71af-47b7-8cc8-b2c02d9961de" (UID: "d6802668-71af-47b7-8cc8-b2c02d9961de"). InnerVolumeSpecName "kube-api-access-swn6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:17.393227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.393191 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6802668-71af-47b7-8cc8-b2c02d9961de-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:17.393227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.393222 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swn6b\" (UniqueName: \"kubernetes.io/projected/d6802668-71af-47b7-8cc8-b2c02d9961de-kube-api-access-swn6b\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:17.919472 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.919435 5476 scope.go:122] "RemoveContainer" containerID="35e8c9b6a752646288344763fe774ca128373ce56b869b45e259bddda3b172a4" Nov 02 15:45:17.919649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.919535 5476 scope.go:122] "RemoveContainer" containerID="fd728d2d956d4f5fa8f6a3469a048a53640a275c6fc696ef6e4e8d00ccba05f9" Nov 02 15:45:17.919706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.919675 5476 scope.go:122] "RemoveContainer" containerID="62953b3f759a5ee9d85c89c628fabc680ed946dfa34ac354c8dbaf12df5dcee6" Nov 02 15:45:17.920678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.920120 5476 scope.go:122] "RemoveContainer" containerID="512542d40ed84def36cbfe9fc3fc55f49c419f7edbbe45e0db91a799d2a0c071" Nov 02 15:45:17.955847 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.955827 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" Nov 02 15:45:17.956309 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.955873 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9" event={"ID":"d6802668-71af-47b7-8cc8-b2c02d9961de","Type":"ContainerDied","Data":"f623cedf908f168a89ea3bf9f082d5d1ad8659b43ad8b390ad01fbb4eeee37f3"} Nov 02 15:45:17.956309 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:17.955939 5476 scope.go:122] "RemoveContainer" containerID="8ec0b5ec94c3e33108d9ee05c95673e2a80befec74331875becd99fd8d911f7d" Nov 02 15:45:18.082295 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.082263 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg"] Nov 02 15:45:18.117035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.116540 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9"] Nov 02 15:45:18.129058 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.129025 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893905-kk4n9"] Nov 02 15:45:18.234500 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.234476 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" Nov 02 15:45:18.303436 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.303403 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-kube-api-access-jsr27\" (UniqueName: \"kubernetes.io/projected/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-kube-api-access-jsr27\") pod \"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a\" (UID: \"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a\") " Nov 02 15:45:18.303632 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.303619 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-maas-api-service-ca\") pod \"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a\" (UID: \"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a\") " Nov 02 15:45:18.304018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.303966 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-maas-api-service-ca" pod "12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" (UID: "12e92d4a-de8d-4244-912c-9b6d2ddf4b8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:18.304122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.304096 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:18.372189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.372150 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-kube-api-access-jsr27" pod "12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" (UID: "12e92d4a-de8d-4244-912c-9b6d2ddf4b8a"). InnerVolumeSpecName "kube-api-access-jsr27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:18.405640 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.405609 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsr27\" (UniqueName: \"kubernetes.io/projected/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a-kube-api-access-jsr27\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:18.915418 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.915389 5476 scope.go:122] "RemoveContainer" containerID="cae9285ae52f0898ac27bf349cb1069130f622fbabf5535fc3377b5552d1a738" Nov 02 15:45:18.967654 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.967611 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" event={"ID":"3ff17841-24b5-4837-9a6c-104783665b81","Type":"ContainerStarted","Data":"9dac0554c0c2f22565ec150fbbecb5e93d2e1b7500df5851f612b03d7e117268"} Nov 02 15:45:18.968970 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.968951 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" Nov 02 15:45:18.969106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.969008 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg" event={"ID":"12e92d4a-de8d-4244-912c-9b6d2ddf4b8a","Type":"ContainerDied","Data":"1ce09f7a94dd9878eeb396064e74359401ca233a1ef1c2968857ab04b1153bf3"} Nov 02 15:45:18.969106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.969069 5476 scope.go:122] "RemoveContainer" containerID="ae89dc5c99e7ae15b949b011ce33a4221727dfdce3edc27cf443cb88f666c0da" Nov 02 15:45:18.971171 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.971148 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" event={"ID":"51a8032f-8a4d-412e-a04f-2c479a8a05df","Type":"ContainerStarted","Data":"2a01cbb7c8df51ffb97ceeab66fe623d106362ead143083b74bde83299487523"} Nov 02 15:45:18.976362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.976333 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" event={"ID":"e884fe81-b9b3-43b9-b873-61603318b613","Type":"ContainerStarted","Data":"0288f03d8db6e5ebcc669a156028ece2fd8f27f72f692ba51a821aed04fea80c"} Nov 02 15:45:18.979346 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:18.979324 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" event={"ID":"79b16007-2c3b-4e4c-b08b-4121b3278a65","Type":"ContainerStarted","Data":"b4cc5cb61a898cb36c66f641eec7673913390745edab310ea9a023d51b3b225a"} Nov 02 15:45:19.414564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.414527 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg"] Nov 02 15:45:19.418975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.418908 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-624908-29893905-wpwjg"] Nov 02 15:45:19.923865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.923828 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" path="/var/lib/kubelet/pods/12e92d4a-de8d-4244-912c-9b6d2ddf4b8a/volumes" Nov 02 15:45:19.924276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.924259 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" path="/var/lib/kubelet/pods/d6802668-71af-47b7-8cc8-b2c02d9961de/volumes" Nov 02 15:45:19.988850 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.988818 5476 generic.go:356] "Generic (PLEG): container finished" podID="3ff17841-24b5-4837-9a6c-104783665b81" containerID="9dac0554c0c2f22565ec150fbbecb5e93d2e1b7500df5851f612b03d7e117268" exitCode=60 Nov 02 15:45:19.989317 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.988890 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" event={"ID":"3ff17841-24b5-4837-9a6c-104783665b81","Type":"ContainerDied","Data":"9dac0554c0c2f22565ec150fbbecb5e93d2e1b7500df5851f612b03d7e117268"} Nov 02 15:45:19.989317 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.988958 5476 scope.go:122] "RemoveContainer" containerID="512542d40ed84def36cbfe9fc3fc55f49c419f7edbbe45e0db91a799d2a0c071" Nov 02 15:45:19.989458 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.989338 5476 scope.go:122] "RemoveContainer" containerID="9dac0554c0c2f22565ec150fbbecb5e93d2e1b7500df5851f612b03d7e117268" Nov 02 15:45:19.989697 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:19.989674 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt_odh-ai-gateway-infra(3ff17841-24b5-4837-9a6c-104783665b81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" podUID="3ff17841-24b5-4837-9a6c-104783665b81" Nov 02 15:45:19.991103 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.991082 5476 generic.go:356] "Generic (PLEG): container finished" podID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerID="2a01cbb7c8df51ffb97ceeab66fe623d106362ead143083b74bde83299487523" exitCode=60 Nov 02 15:45:19.991204 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.991155 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" event={"ID":"51a8032f-8a4d-412e-a04f-2c479a8a05df","Type":"ContainerDied","Data":"2a01cbb7c8df51ffb97ceeab66fe623d106362ead143083b74bde83299487523"} Nov 02 15:45:19.991503 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.991487 5476 scope.go:122] "RemoveContainer" containerID="2a01cbb7c8df51ffb97ceeab66fe623d106362ead143083b74bde83299487523" Nov 02 15:45:19.991769 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:19.991744 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x_odh-ai-gateway-infra(51a8032f-8a4d-412e-a04f-2c479a8a05df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" Nov 02 15:45:19.992953 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.992934 5476 generic.go:356] "Generic (PLEG): container finished" podID="e884fe81-b9b3-43b9-b873-61603318b613" containerID="0288f03d8db6e5ebcc669a156028ece2fd8f27f72f692ba51a821aed04fea80c" exitCode=60 Nov 02 15:45:19.993059 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.993030 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" event={"ID":"e884fe81-b9b3-43b9-b873-61603318b613","Type":"ContainerDied","Data":"0288f03d8db6e5ebcc669a156028ece2fd8f27f72f692ba51a821aed04fea80c"} Nov 02 15:45:19.993408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.993390 5476 scope.go:122] "RemoveContainer" containerID="0288f03d8db6e5ebcc669a156028ece2fd8f27f72f692ba51a821aed04fea80c" Nov 02 15:45:19.993662 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:19.993641 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj_odh-ai-gateway-infra(e884fe81-b9b3-43b9-b873-61603318b613)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" podUID="e884fe81-b9b3-43b9-b873-61603318b613" Nov 02 15:45:19.994830 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.994813 5476 generic.go:356] "Generic (PLEG): container finished" podID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerID="b4cc5cb61a898cb36c66f641eec7673913390745edab310ea9a023d51b3b225a" exitCode=60 Nov 02 15:45:19.994925 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.994884 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" event={"ID":"79b16007-2c3b-4e4c-b08b-4121b3278a65","Type":"ContainerDied","Data":"b4cc5cb61a898cb36c66f641eec7673913390745edab310ea9a023d51b3b225a"} Nov 02 15:45:19.995243 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.995228 5476 scope.go:122] "RemoveContainer" containerID="b4cc5cb61a898cb36c66f641eec7673913390745edab310ea9a023d51b3b225a" Nov 02 15:45:19.995457 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:19.995436 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2_odh-ai-gateway-infra(79b16007-2c3b-4e4c-b08b-4121b3278a65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" Nov 02 15:45:19.997023 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.996985 5476 generic.go:356] "Generic (PLEG): container finished" podID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerID="00dbc53dfa47b38bbe2c168cf5354a643195a10e41ba54638bdfffaa2553aca3" exitCode=60 Nov 02 15:45:19.997107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.997037 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" event={"ID":"3e5c73d7-b426-48b3-bd36-3f628e431f85","Type":"ContainerDied","Data":"00dbc53dfa47b38bbe2c168cf5354a643195a10e41ba54638bdfffaa2553aca3"} Nov 02 15:45:19.997344 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:19.997329 5476 scope.go:122] "RemoveContainer" containerID="00dbc53dfa47b38bbe2c168cf5354a643195a10e41ba54638bdfffaa2553aca3" Nov 02 15:45:19.997535 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:19.997515 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk_odh-ai-gateway-infra(3e5c73d7-b426-48b3-bd36-3f628e431f85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" Nov 02 15:45:20.002101 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:20.002085 5476 scope.go:122] "RemoveContainer" containerID="fd728d2d956d4f5fa8f6a3469a048a53640a275c6fc696ef6e4e8d00ccba05f9" Nov 02 15:45:20.014537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:20.014517 5476 scope.go:122] "RemoveContainer" containerID="35e8c9b6a752646288344763fe774ca128373ce56b869b45e259bddda3b172a4" Nov 02 15:45:20.027323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:20.027305 5476 scope.go:122] "RemoveContainer" containerID="62953b3f759a5ee9d85c89c628fabc680ed946dfa34ac354c8dbaf12df5dcee6" Nov 02 15:45:20.043205 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:20.043178 5476 scope.go:122] "RemoveContainer" containerID="cae9285ae52f0898ac27bf349cb1069130f622fbabf5535fc3377b5552d1a738" Nov 02 15:45:20.091873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:20.091839 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt"] Nov 02 15:45:20.146473 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:20.146437 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:45:20.204521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:20.204490 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj"] Nov 02 15:45:20.310511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:20.310428 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2"] Nov 02 15:45:20.541623 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:20.541591 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x"] Nov 02 15:45:21.222218 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.214222 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" Nov 02 15:45:21.291457 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.290710 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" Nov 02 15:45:21.295842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.295814 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" Nov 02 15:45:21.321170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.321151 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" Nov 02 15:45:21.343505 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.343480 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e884fe81-b9b3-43b9-b873-61603318b613-kube-api-access-b8s8v\" (UniqueName: \"kubernetes.io/projected/e884fe81-b9b3-43b9-b873-61603318b613-kube-api-access-b8s8v\") pod \"e884fe81-b9b3-43b9-b873-61603318b613\" (UID: \"e884fe81-b9b3-43b9-b873-61603318b613\") " Nov 02 15:45:21.343630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.343604 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e884fe81-b9b3-43b9-b873-61603318b613-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e884fe81-b9b3-43b9-b873-61603318b613-maas-api-service-ca\") pod \"e884fe81-b9b3-43b9-b873-61603318b613\" (UID: \"e884fe81-b9b3-43b9-b873-61603318b613\") " Nov 02 15:45:21.343940 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.343917 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e884fe81-b9b3-43b9-b873-61603318b613-maas-api-service-ca" pod "e884fe81-b9b3-43b9-b873-61603318b613" (UID: "e884fe81-b9b3-43b9-b873-61603318b613"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:21.345478 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.345459 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e884fe81-b9b3-43b9-b873-61603318b613-kube-api-access-b8s8v" pod "e884fe81-b9b3-43b9-b873-61603318b613" (UID: "e884fe81-b9b3-43b9-b873-61603318b613"). InnerVolumeSpecName "kube-api-access-b8s8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:21.444531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.444490 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b16007-2c3b-4e4c-b08b-4121b3278a65-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b16007-2c3b-4e4c-b08b-4121b3278a65-maas-api-service-ca\") pod \"79b16007-2c3b-4e4c-b08b-4121b3278a65\" (UID: \"79b16007-2c3b-4e4c-b08b-4121b3278a65\") " Nov 02 15:45:21.444531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.444538 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b16007-2c3b-4e4c-b08b-4121b3278a65-kube-api-access-2g22q\" (UniqueName: \"kubernetes.io/projected/79b16007-2c3b-4e4c-b08b-4121b3278a65-kube-api-access-2g22q\") pod \"79b16007-2c3b-4e4c-b08b-4121b3278a65\" (UID: \"79b16007-2c3b-4e4c-b08b-4121b3278a65\") " Nov 02 15:45:21.444813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.444617 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a8032f-8a4d-412e-a04f-2c479a8a05df-kube-api-access-2z5nj\" (UniqueName: \"kubernetes.io/projected/51a8032f-8a4d-412e-a04f-2c479a8a05df-kube-api-access-2z5nj\") pod \"51a8032f-8a4d-412e-a04f-2c479a8a05df\" (UID: \"51a8032f-8a4d-412e-a04f-2c479a8a05df\") " Nov 02 15:45:21.444813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.444762 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff17841-24b5-4837-9a6c-104783665b81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff17841-24b5-4837-9a6c-104783665b81-maas-api-service-ca\") pod \"3ff17841-24b5-4837-9a6c-104783665b81\" (UID: \"3ff17841-24b5-4837-9a6c-104783665b81\") " Nov 02 15:45:21.444813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.444803 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51a8032f-8a4d-412e-a04f-2c479a8a05df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51a8032f-8a4d-412e-a04f-2c479a8a05df-maas-api-service-ca\") pod \"51a8032f-8a4d-412e-a04f-2c479a8a05df\" (UID: \"51a8032f-8a4d-412e-a04f-2c479a8a05df\") " Nov 02 15:45:21.444987 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.444858 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b16007-2c3b-4e4c-b08b-4121b3278a65-maas-api-service-ca" pod "79b16007-2c3b-4e4c-b08b-4121b3278a65" (UID: "79b16007-2c3b-4e4c-b08b-4121b3278a65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:21.444987 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.444887 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff17841-24b5-4837-9a6c-104783665b81-kube-api-access-prq2k\" (UniqueName: \"kubernetes.io/projected/3ff17841-24b5-4837-9a6c-104783665b81-kube-api-access-prq2k\") pod \"3ff17841-24b5-4837-9a6c-104783665b81\" (UID: \"3ff17841-24b5-4837-9a6c-104783665b81\") " Nov 02 15:45:21.445171 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.445149 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff17841-24b5-4837-9a6c-104783665b81-maas-api-service-ca" pod "3ff17841-24b5-4837-9a6c-104783665b81" (UID: "3ff17841-24b5-4837-9a6c-104783665b81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:21.445275 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.445252 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51a8032f-8a4d-412e-a04f-2c479a8a05df-maas-api-service-ca" pod "51a8032f-8a4d-412e-a04f-2c479a8a05df" (UID: "51a8032f-8a4d-412e-a04f-2c479a8a05df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:21.445460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.445438 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ff17841-24b5-4837-9a6c-104783665b81-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.445460 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.445460 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51a8032f-8a4d-412e-a04f-2c479a8a05df-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.445662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.445472 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e884fe81-b9b3-43b9-b873-61603318b613-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.445662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.445486 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b16007-2c3b-4e4c-b08b-4121b3278a65-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.445662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.445495 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8s8v\" (UniqueName: \"kubernetes.io/projected/e884fe81-b9b3-43b9-b873-61603318b613-kube-api-access-b8s8v\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.446882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.446861 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b16007-2c3b-4e4c-b08b-4121b3278a65-kube-api-access-2g22q" pod "79b16007-2c3b-4e4c-b08b-4121b3278a65" (UID: "79b16007-2c3b-4e4c-b08b-4121b3278a65"). InnerVolumeSpecName "kube-api-access-2g22q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:21.447374 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.447358 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff17841-24b5-4837-9a6c-104783665b81-kube-api-access-prq2k" pod "3ff17841-24b5-4837-9a6c-104783665b81" (UID: "3ff17841-24b5-4837-9a6c-104783665b81"). InnerVolumeSpecName "kube-api-access-prq2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:21.447458 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.447445 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a8032f-8a4d-412e-a04f-2c479a8a05df-kube-api-access-2z5nj" pod "51a8032f-8a4d-412e-a04f-2c479a8a05df" (UID: "51a8032f-8a4d-412e-a04f-2c479a8a05df"). InnerVolumeSpecName "kube-api-access-2z5nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:21.546921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.546826 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z5nj\" (UniqueName: \"kubernetes.io/projected/51a8032f-8a4d-412e-a04f-2c479a8a05df-kube-api-access-2z5nj\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.546921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.546855 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prq2k\" (UniqueName: \"kubernetes.io/projected/3ff17841-24b5-4837-9a6c-104783665b81-kube-api-access-prq2k\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.546921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.546868 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g22q\" (UniqueName: \"kubernetes.io/projected/79b16007-2c3b-4e4c-b08b-4121b3278a65-kube-api-access-2g22q\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.572225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.572189 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk"] Nov 02 15:45:21.718234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.718204 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" Nov 02 15:45:21.851489 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.851399 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5c73d7-b426-48b3-bd36-3f628e431f85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5c73d7-b426-48b3-bd36-3f628e431f85-maas-api-service-ca\") pod \"3e5c73d7-b426-48b3-bd36-3f628e431f85\" (UID: \"3e5c73d7-b426-48b3-bd36-3f628e431f85\") " Nov 02 15:45:21.851676 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.851497 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5c73d7-b426-48b3-bd36-3f628e431f85-kube-api-access-txwvr\" (UniqueName: \"kubernetes.io/projected/3e5c73d7-b426-48b3-bd36-3f628e431f85-kube-api-access-txwvr\") pod \"3e5c73d7-b426-48b3-bd36-3f628e431f85\" (UID: \"3e5c73d7-b426-48b3-bd36-3f628e431f85\") " Nov 02 15:45:21.851760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.851723 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5c73d7-b426-48b3-bd36-3f628e431f85-maas-api-service-ca" pod "3e5c73d7-b426-48b3-bd36-3f628e431f85" (UID: "3e5c73d7-b426-48b3-bd36-3f628e431f85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:21.851926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.851913 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5c73d7-b426-48b3-bd36-3f628e431f85-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.853501 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.853480 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5c73d7-b426-48b3-bd36-3f628e431f85-kube-api-access-txwvr" pod "3e5c73d7-b426-48b3-bd36-3f628e431f85" (UID: "3e5c73d7-b426-48b3-bd36-3f628e431f85"). InnerVolumeSpecName "kube-api-access-txwvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:21.953423 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:21.953387 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txwvr\" (UniqueName: \"kubernetes.io/projected/3e5c73d7-b426-48b3-bd36-3f628e431f85-kube-api-access-txwvr\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:22.090921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.090890 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" Nov 02 15:45:22.091123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.090888 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk" event={"ID":"3e5c73d7-b426-48b3-bd36-3f628e431f85","Type":"ContainerDied","Data":"48a24e7d86b47256ce0c4720ca29ad8da48f9c6a78b899e5a9452ce893115bb5"} Nov 02 15:45:22.091205 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.091180 5476 scope.go:122] "RemoveContainer" containerID="00dbc53dfa47b38bbe2c168cf5354a643195a10e41ba54638bdfffaa2553aca3" Nov 02 15:45:22.093271 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.093253 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" Nov 02 15:45:22.093627 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.093292 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt" event={"ID":"3ff17841-24b5-4837-9a6c-104783665b81","Type":"ContainerDied","Data":"08b9f3cb8ceba711c0e692cac717e236005a2942ef8855833e30e891c78968b6"} Nov 02 15:45:22.094947 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.094922 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" event={"ID":"51a8032f-8a4d-412e-a04f-2c479a8a05df","Type":"ContainerDied","Data":"d10ef12d97ff980211f586307bb610eb241a80a6bec6472339cc17361143e21d"} Nov 02 15:45:22.095069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.094946 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x" Nov 02 15:45:22.097701 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.097678 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" Nov 02 15:45:22.097808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.097701 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj" event={"ID":"e884fe81-b9b3-43b9-b873-61603318b613","Type":"ContainerDied","Data":"e097ee1592f7e5d9513e0cf471703a6ef9fea27df00b703aff4083f785642764"} Nov 02 15:45:22.099567 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.099533 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" event={"ID":"79b16007-2c3b-4e4c-b08b-4121b3278a65","Type":"ContainerDied","Data":"8bb499fa6be03d5b76c1d60faf704a3aedc7ea74783dc2a3b8378c051b50eadd"} Nov 02 15:45:22.099643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.099541 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2" Nov 02 15:45:22.105048 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.105033 5476 scope.go:122] "RemoveContainer" containerID="9dac0554c0c2f22565ec150fbbecb5e93d2e1b7500df5851f612b03d7e117268" Nov 02 15:45:22.116810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.116788 5476 scope.go:122] "RemoveContainer" containerID="2a01cbb7c8df51ffb97ceeab66fe623d106362ead143083b74bde83299487523" Nov 02 15:45:22.126054 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.126032 5476 scope.go:122] "RemoveContainer" containerID="0288f03d8db6e5ebcc669a156028ece2fd8f27f72f692ba51a821aed04fea80c" Nov 02 15:45:22.134793 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.134771 5476 scope.go:122] "RemoveContainer" containerID="b4cc5cb61a898cb36c66f641eec7673913390745edab310ea9a023d51b3b225a" Nov 02 15:45:22.249677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.249633 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x"] Nov 02 15:45:22.254201 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.254171 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ff6e1f-29893905-7h64x"] Nov 02 15:45:22.356464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.356377 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj"] Nov 02 15:45:22.363499 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.363471 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893905-mxtjj"] Nov 02 15:45:22.470430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.470398 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2"] Nov 02 15:45:22.476050 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.476021 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eb6bd4-29893905-htkz2"] Nov 02 15:45:22.578321 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.578289 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk"] Nov 02 15:45:22.582554 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.582526 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-b1ee37-29893905-fkmmk"] Nov 02 15:45:22.733888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.733810 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt"] Nov 02 15:45:22.755377 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:22.755344 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-f6b245-29893905-n9zvt"] Nov 02 15:45:23.922857 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:23.922825 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" path="/var/lib/kubelet/pods/3e5c73d7-b426-48b3-bd36-3f628e431f85/volumes" Nov 02 15:45:23.923359 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:23.923290 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff17841-24b5-4837-9a6c-104783665b81" path="/var/lib/kubelet/pods/3ff17841-24b5-4837-9a6c-104783665b81/volumes" Nov 02 15:45:23.923646 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:23.923632 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" path="/var/lib/kubelet/pods/51a8032f-8a4d-412e-a04f-2c479a8a05df/volumes" Nov 02 15:45:23.924307 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:23.924292 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" path="/var/lib/kubelet/pods/79b16007-2c3b-4e4c-b08b-4121b3278a65/volumes" Nov 02 15:45:23.924659 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:23.924647 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e884fe81-b9b3-43b9-b873-61603318b613" path="/var/lib/kubelet/pods/e884fe81-b9b3-43b9-b873-61603318b613/volumes" Nov 02 15:45:30.922127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:30.922099 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4"] Nov 02 15:45:30.922575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:30.922290 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" containerID="cri-o://ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7" gracePeriod=30 Nov 02 15:45:33.132727 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.132682 5476 patch_prober.go:29] interesting pod/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:45:33.133425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.132768 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:45:33.461171 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461085 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph"] Nov 02 15:45:33.461860 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461837 5476 cpu_manager.go:414] "containerMap: removing container" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerName="cleanup" Nov 02 15:45:33.461909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461865 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerName="cleanup" Nov 02 15:45:33.461909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461878 5476 cpu_manager.go:414] "containerMap: removing container" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerName="cleanup" Nov 02 15:45:33.461909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461887 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerName="cleanup" Nov 02 15:45:33.462021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461911 5476 cpu_manager.go:414] "containerMap: removing container" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerName="cleanup" Nov 02 15:45:33.462021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461919 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerName="cleanup" Nov 02 15:45:33.462021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461932 5476 cpu_manager.go:414] "containerMap: removing container" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerName="cleanup" Nov 02 15:45:33.462021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461940 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerName="cleanup" Nov 02 15:45:33.462021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461957 5476 cpu_manager.go:414] "containerMap: removing container" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerName="cleanup" Nov 02 15:45:33.462021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461964 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerName="cleanup" Nov 02 15:45:33.462021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461980 5476 cpu_manager.go:414] "containerMap: removing container" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerName="cleanup" Nov 02 15:45:33.462021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.461986 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerName="cleanup" Nov 02 15:45:33.462021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462011 5476 cpu_manager.go:414] "containerMap: removing container" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerName="cleanup" Nov 02 15:45:33.462021 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462018 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462035 5476 cpu_manager.go:414] "containerMap: removing container" podUID="3ff17841-24b5-4837-9a6c-104783665b81" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462043 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff17841-24b5-4837-9a6c-104783665b81" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462054 5476 cpu_manager.go:414] "containerMap: removing container" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462060 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462073 5476 cpu_manager.go:414] "containerMap: removing container" podUID="e884fe81-b9b3-43b9-b873-61603318b613" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462080 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e884fe81-b9b3-43b9-b873-61603318b613" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462106 5476 cpu_manager.go:414] "containerMap: removing container" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462113 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462122 5476 cpu_manager.go:414] "containerMap: removing container" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462128 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462143 5476 cpu_manager.go:414] "containerMap: removing container" podUID="3ff17841-24b5-4837-9a6c-104783665b81" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462150 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff17841-24b5-4837-9a6c-104783665b81" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462166 5476 cpu_manager.go:414] "containerMap: removing container" podUID="e884fe81-b9b3-43b9-b873-61603318b613" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462174 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e884fe81-b9b3-43b9-b873-61603318b613" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462184 5476 cpu_manager.go:414] "containerMap: removing container" podUID="e884fe81-b9b3-43b9-b873-61603318b613" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462191 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e884fe81-b9b3-43b9-b873-61603318b613" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462207 5476 cpu_manager.go:414] "containerMap: removing container" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462215 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462228 5476 cpu_manager.go:414] "containerMap: removing container" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerName="cleanup" Nov 02 15:45:33.462363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462235 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462381 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462392 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462404 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff17841-24b5-4837-9a6c-104783665b81" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462412 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462422 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e884fe81-b9b3-43b9-b873-61603318b613" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462431 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462441 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462451 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff17841-24b5-4837-9a6c-104783665b81" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462460 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462468 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e884fe81-b9b3-43b9-b873-61603318b613" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462476 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462484 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e884fe81-b9b3-43b9-b873-61603318b613" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462493 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462503 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462511 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff17841-24b5-4837-9a6c-104783665b81" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462522 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462531 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerName="cleanup" Nov 02 15:45:33.462965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.462538 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerName="cleanup" Nov 02 15:45:33.468322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.468299 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:33.503186 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.503152 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03db5bb4-rhsph\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:33.503455 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.503426 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xkdc\" (UniqueName: \"kubernetes.io/projected/b8777e2c-5138-42a2-88ea-2426d7de5ea1-kube-api-access-4xkdc\") pod \"maas-api-revoke-sub-03db5bb4-rhsph\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:33.503537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.503484 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03db5bb4-rhsph\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:33.521871 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.521839 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph"] Nov 02 15:45:33.563439 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.563369 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:45:33.563637 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.563453 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:45:33.605051 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.605015 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xkdc\" (UniqueName: \"kubernetes.io/projected/b8777e2c-5138-42a2-88ea-2426d7de5ea1-kube-api-access-4xkdc\") pod \"maas-api-revoke-sub-03db5bb4-rhsph\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:33.605051 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.605065 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03db5bb4-rhsph\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:33.605299 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.605118 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03db5bb4-rhsph\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:33.605896 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.605876 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03db5bb4-rhsph\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:33.607500 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.607480 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03db5bb4-rhsph\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:33.663242 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.663212 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xkdc\" (UniqueName: \"kubernetes.io/projected/b8777e2c-5138-42a2-88ea-2426d7de5ea1-kube-api-access-4xkdc\") pod \"maas-api-revoke-sub-03db5bb4-rhsph\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:33.782810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:33.782725 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:34.157060 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:34.157026 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph"] Nov 02 15:45:34.162536 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:45:34.162507 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8777e2c_5138_42a2_88ea_2426d7de5ea1.slice/crio-a91ce252980b62475f5307a48fee0d467213df231f6fab4a7508169da85dd317 WatchSource:0}: Error finding container a91ce252980b62475f5307a48fee0d467213df231f6fab4a7508169da85dd317: Status 404 returned error can't find the container with id a91ce252980b62475f5307a48fee0d467213df231f6fab4a7508169da85dd317 Nov 02 15:45:34.199729 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:34.199699 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" event={"ID":"b8777e2c-5138-42a2-88ea-2426d7de5ea1","Type":"ContainerStarted","Data":"a91ce252980b62475f5307a48fee0d467213df231f6fab4a7508169da85dd317"} Nov 02 15:45:35.207360 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:35.207322 5476 generic.go:356] "Generic (PLEG): container finished" podID="b8777e2c-5138-42a2-88ea-2426d7de5ea1" containerID="f879803b9b4e17806763edc057f15dcbbf83ab4ad7ea97eacc3df932ca29a462" exitCode=0 Nov 02 15:45:35.207754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:35.207410 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" event={"ID":"b8777e2c-5138-42a2-88ea-2426d7de5ea1","Type":"ContainerDied","Data":"f879803b9b4e17806763edc057f15dcbbf83ab4ad7ea97eacc3df932ca29a462"} Nov 02 15:45:36.192121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.192055 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:45:36.215450 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.215412 5476 generic.go:356] "Generic (PLEG): container finished" podID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerID="ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7" exitCode=0 Nov 02 15:45:36.215913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.215473 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" Nov 02 15:45:36.215913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.215497 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" event={"ID":"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b","Type":"ContainerDied","Data":"ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7"} Nov 02 15:45:36.215913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.215566 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4" event={"ID":"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b","Type":"ContainerDied","Data":"81a70289fa5f37906c560bafa9ee7f9789f3323b66672359da80d8a6787b0961"} Nov 02 15:45:36.215913 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.215599 5476 scope.go:122] "RemoveContainer" containerID="ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7" Nov 02 15:45:36.226124 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.226101 5476 scope.go:122] "RemoveContainer" containerID="726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00" Nov 02 15:45:36.236597 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.236565 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istiod-ca-cert\") pod \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " Nov 02 15:45:36.236768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.236609 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-kube-api-access-4f4m6\" (UniqueName: \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-kube-api-access-4f4m6\") pod \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " Nov 02 15:45:36.236768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.236645 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-envoy\") pod \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " Nov 02 15:45:36.236880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.236768 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-certs\") pod \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " Nov 02 15:45:36.236880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.236850 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-credential-socket\") pod \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " Nov 02 15:45:36.236988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.236897 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-socket\") pod \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " Nov 02 15:45:36.236988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.236947 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-token\" (UniqueName: \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-token\") pod \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " Nov 02 15:45:36.236988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.236952 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istiod-ca-cert" pod "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" (UID: "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:36.237170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.237016 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-podinfo\") pod \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " Nov 02 15:45:36.237170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.237111 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-data\") pod \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\" (UID: \"14d3cf79-d7b4-4d45-8d6a-bdc99a46531b\") " Nov 02 15:45:36.237170 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.237150 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-certs" pod "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" (UID: "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:36.237566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.237541 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.237566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.237566 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.237805 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.237609 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-socket" pod "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" (UID: "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:36.237805 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.237637 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-data" pod "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" (UID: "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:36.238042 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.237823 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-credential-socket" pod "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" (UID: "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:36.238953 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.238924 5476 scope.go:122] "RemoveContainer" containerID="ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7" Nov 02 15:45:36.239126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.239102 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-envoy" pod "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" (UID: "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:36.239474 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:36.239445 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7\": container with ID starting with ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7 not found: ID does not exist" containerID="ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7" Nov 02 15:45:36.239544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.239487 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7"} err="failed to get container status \"ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7\": rpc error: code = NotFound desc = could not find container \"ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7\": container with ID starting with ba0941dff900926b40f0437d678e5c9bac09b1ae75bc608772539593087807e7 not found: ID does not exist" Nov 02 15:45:36.239544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.239513 5476 scope.go:122] "RemoveContainer" containerID="726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00" Nov 02 15:45:36.239640 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.239612 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-kube-api-access-4f4m6" pod "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" (UID: "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b"). InnerVolumeSpecName "kube-api-access-4f4m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:36.239697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.239681 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-podinfo" pod "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" (UID: "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:45:36.239757 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.239738 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-token" pod "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" (UID: "14d3cf79-d7b4-4d45-8d6a-bdc99a46531b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:36.239876 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:36.239857 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00\": container with ID starting with 726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00 not found: ID does not exist" containerID="726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00" Nov 02 15:45:36.239922 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.239884 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00"} err="failed to get container status \"726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00\": rpc error: code = NotFound desc = could not find container \"726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00\": container with ID starting with 726d0b20dad2f129f8ca11f0c2664a3d1d2b7e36b10a664b23363fc1a4106f00 not found: ID does not exist" Nov 02 15:45:36.339101 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.339058 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.339101 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.339102 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.339362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.339131 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.339362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.339146 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.339362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.339160 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.339362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.339179 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f4m6\" (UniqueName: \"kubernetes.io/projected/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-kube-api-access-4f4m6\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.339362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.339203 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.361437 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.361405 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:36.440556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.440524 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-openshift-ca\") pod \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " Nov 02 15:45:36.440747 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.440593 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8777e2c-5138-42a2-88ea-2426d7de5ea1-kube-api-access-4xkdc\" (UniqueName: \"kubernetes.io/projected/b8777e2c-5138-42a2-88ea-2426d7de5ea1-kube-api-access-4xkdc\") pod \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " Nov 02 15:45:36.440747 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.440631 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-xks-ca\") pod \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\" (UID: \"b8777e2c-5138-42a2-88ea-2426d7de5ea1\") " Nov 02 15:45:36.440943 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.440919 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-openshift-ca" pod "b8777e2c-5138-42a2-88ea-2426d7de5ea1" (UID: "b8777e2c-5138-42a2-88ea-2426d7de5ea1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:36.441231 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.441208 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.442841 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.442790 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-xks-ca" pod "b8777e2c-5138-42a2-88ea-2426d7de5ea1" (UID: "b8777e2c-5138-42a2-88ea-2426d7de5ea1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:45:36.442841 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.442811 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8777e2c-5138-42a2-88ea-2426d7de5ea1-kube-api-access-4xkdc" pod "b8777e2c-5138-42a2-88ea-2426d7de5ea1" (UID: "b8777e2c-5138-42a2-88ea-2426d7de5ea1"). InnerVolumeSpecName "kube-api-access-4xkdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:36.542273 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.542242 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xkdc\" (UniqueName: \"kubernetes.io/projected/b8777e2c-5138-42a2-88ea-2426d7de5ea1-kube-api-access-4xkdc\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.542273 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.542274 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8777e2c-5138-42a2-88ea-2426d7de5ea1-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:36.694607 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.694522 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4"] Nov 02 15:45:36.701706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:36.701673 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9ca2cf-openshift-default-547cbb69bf-5ncw4"] Nov 02 15:45:37.224063 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:37.224028 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" event={"ID":"b8777e2c-5138-42a2-88ea-2426d7de5ea1","Type":"ContainerDied","Data":"a91ce252980b62475f5307a48fee0d467213df231f6fab4a7508169da85dd317"} Nov 02 15:45:37.224063 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:37.224049 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph" Nov 02 15:45:37.224614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:37.224087 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a91ce252980b62475f5307a48fee0d467213df231f6fab4a7508169da85dd317" Nov 02 15:45:37.923156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:37.923116 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" path="/var/lib/kubelet/pods/14d3cf79-d7b4-4d45-8d6a-bdc99a46531b/volumes" Nov 02 15:45:50.147135 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:45:50.147096 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:45:58.227203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:45:58.227158 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:46:20.148270 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:46:20.148230 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:46:25.719752 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:25.719719 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg"] Nov 02 15:46:30.720097 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:30.719978 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" podUID="bd84d132-0deb-4c37-943d-1bb1a411fb41" containerName="payload-pre-processing" containerID="cri-o://084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8" gracePeriod=25 Nov 02 15:46:35.436157 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:35.436116 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648"] Nov 02 15:46:35.436690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:35.436355 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" containerID="cri-o://ec5ecba1676c54fcfa5f68b065d05a1b21382beaa3df696be4ef996eccc4babb" gracePeriod=30 Nov 02 15:46:35.579414 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:35.579371 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-75648 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:46:35.579588 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:35.579440 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:46:39.813017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:39.812961 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" podUID="bd84d132-0deb-4c37-943d-1bb1a411fb41" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:46:40.397610 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.397567 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:46:40.633269 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.633236 5476 generic.go:356] "Generic (PLEG): container finished" podID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerID="ec5ecba1676c54fcfa5f68b065d05a1b21382beaa3df696be4ef996eccc4babb" exitCode=0 Nov 02 15:46:40.633470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.633377 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" event={"ID":"5eda8baf-c3cb-40ca-9acc-219484ff4dd7","Type":"ContainerDied","Data":"ec5ecba1676c54fcfa5f68b065d05a1b21382beaa3df696be4ef996eccc4babb"} Nov 02 15:46:40.738337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.738305 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:46:40.779655 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.779617 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-data\") pod \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " Nov 02 15:46:40.779851 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.779659 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-certs\") pod \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " Nov 02 15:46:40.779851 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.779723 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-kube-api-access-f4gph\" (UniqueName: \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-kube-api-access-f4gph\") pod \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " Nov 02 15:46:40.779851 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.779807 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istiod-ca-cert\") pod \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " Nov 02 15:46:40.779989 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.779902 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-credential-socket\") pod \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " Nov 02 15:46:40.779989 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.779959 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-data" pod "5eda8baf-c3cb-40ca-9acc-219484ff4dd7" (UID: "5eda8baf-c3cb-40ca-9acc-219484ff4dd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.779989 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.779965 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-envoy\") pod \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " Nov 02 15:46:40.780137 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.779979 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-certs" pod "5eda8baf-c3cb-40ca-9acc-219484ff4dd7" (UID: "5eda8baf-c3cb-40ca-9acc-219484ff4dd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.780137 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.780042 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-token\" (UniqueName: \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-token\") pod \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " Nov 02 15:46:40.780137 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.780127 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-podinfo\") pod \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " Nov 02 15:46:40.780289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.780186 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-socket\") pod \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\" (UID: \"5eda8baf-c3cb-40ca-9acc-219484ff4dd7\") " Nov 02 15:46:40.780289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.780229 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istiod-ca-cert" pod "5eda8baf-c3cb-40ca-9acc-219484ff4dd7" (UID: "5eda8baf-c3cb-40ca-9acc-219484ff4dd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:40.780289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.780268 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-credential-socket" pod "5eda8baf-c3cb-40ca-9acc-219484ff4dd7" (UID: "5eda8baf-c3cb-40ca-9acc-219484ff4dd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.780565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.780542 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-socket" pod "5eda8baf-c3cb-40ca-9acc-219484ff4dd7" (UID: "5eda8baf-c3cb-40ca-9acc-219484ff4dd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.780693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.780676 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.780761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.780699 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.780761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.780717 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.780761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.780732 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.780761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.780744 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.782240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.782212 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-kube-api-access-f4gph" pod "5eda8baf-c3cb-40ca-9acc-219484ff4dd7" (UID: "5eda8baf-c3cb-40ca-9acc-219484ff4dd7"). InnerVolumeSpecName "kube-api-access-f4gph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:40.782583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.782557 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-envoy" pod "5eda8baf-c3cb-40ca-9acc-219484ff4dd7" (UID: "5eda8baf-c3cb-40ca-9acc-219484ff4dd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:40.782667 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.782583 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-podinfo" pod "5eda8baf-c3cb-40ca-9acc-219484ff4dd7" (UID: "5eda8baf-c3cb-40ca-9acc-219484ff4dd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:46:40.783129 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.783104 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-token" pod "5eda8baf-c3cb-40ca-9acc-219484ff4dd7" (UID: "5eda8baf-c3cb-40ca-9acc-219484ff4dd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:40.881733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.881690 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4gph\" (UniqueName: \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-kube-api-access-f4gph\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.881733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.881732 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.882333 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.881747 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:40.882333 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:40.881762 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eda8baf-c3cb-40ca-9acc-219484ff4dd7-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.641313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:41.641273 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" Nov 02 15:46:41.641313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:41.641284 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648" event={"ID":"5eda8baf-c3cb-40ca-9acc-219484ff4dd7","Type":"ContainerDied","Data":"fa2a2688638634b6362c89ca0c577e1323ff209637b8ec84f67cd0d398e75a96"} Nov 02 15:46:41.641556 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:41.641379 5476 scope.go:122] "RemoveContainer" containerID="ec5ecba1676c54fcfa5f68b065d05a1b21382beaa3df696be4ef996eccc4babb" Nov 02 15:46:41.796191 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:41.796151 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648"] Nov 02 15:46:41.842239 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:41.842203 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-75648"] Nov 02 15:46:41.923755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:41.923674 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" path="/var/lib/kubelet/pods/5eda8baf-c3cb-40ca-9acc-219484ff4dd7/volumes" Nov 02 15:46:49.647425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:49.647390 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29"] Nov 02 15:46:49.691043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:49.690980 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3b351-52h29"] Nov 02 15:46:49.812396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:49.812362 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" podUID="bd84d132-0deb-4c37-943d-1bb1a411fb41" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:46:49.923736 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:49.923659 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53424310-bc1b-4988-b546-00a55039a9ec" path="/var/lib/kubelet/pods/53424310-bc1b-4988-b546-00a55039a9ec/volumes" Nov 02 15:46:50.147159 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:46:50.147119 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:46:50.987210 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:50.987144 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:46:51.102529 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.102494 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd84d132-0deb-4c37-943d-1bb1a411fb41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd84d132-0deb-4c37-943d-1bb1a411fb41-plugins-config-volume\") pod \"bd84d132-0deb-4c37-943d-1bb1a411fb41\" (UID: \"bd84d132-0deb-4c37-943d-1bb1a411fb41\") " Nov 02 15:46:51.102731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.102654 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd84d132-0deb-4c37-943d-1bb1a411fb41-kube-api-access-btv74\" (UniqueName: \"kubernetes.io/projected/bd84d132-0deb-4c37-943d-1bb1a411fb41-kube-api-access-btv74\") pod \"bd84d132-0deb-4c37-943d-1bb1a411fb41\" (UID: \"bd84d132-0deb-4c37-943d-1bb1a411fb41\") " Nov 02 15:46:51.102936 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.102908 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd84d132-0deb-4c37-943d-1bb1a411fb41-plugins-config-volume" pod "bd84d132-0deb-4c37-943d-1bb1a411fb41" (UID: "bd84d132-0deb-4c37-943d-1bb1a411fb41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:51.103167 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.103140 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd84d132-0deb-4c37-943d-1bb1a411fb41-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:51.104811 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.104778 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd84d132-0deb-4c37-943d-1bb1a411fb41-kube-api-access-btv74" pod "bd84d132-0deb-4c37-943d-1bb1a411fb41" (UID: "bd84d132-0deb-4c37-943d-1bb1a411fb41"). InnerVolumeSpecName "kube-api-access-btv74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:51.204406 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.204361 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btv74\" (UniqueName: \"kubernetes.io/projected/bd84d132-0deb-4c37-943d-1bb1a411fb41-kube-api-access-btv74\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:51.718546 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.718507 5476 generic.go:356] "Generic (PLEG): container finished" podID="bd84d132-0deb-4c37-943d-1bb1a411fb41" containerID="084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8" exitCode=0 Nov 02 15:46:51.718775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.718566 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" Nov 02 15:46:51.718775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.718598 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" event={"ID":"bd84d132-0deb-4c37-943d-1bb1a411fb41","Type":"ContainerDied","Data":"084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8"} Nov 02 15:46:51.718775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.718692 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg" event={"ID":"bd84d132-0deb-4c37-943d-1bb1a411fb41","Type":"ContainerDied","Data":"220838774e7d52bf70e5a8ac6c14e0f89d070689592bac5a880d48fb65559309"} Nov 02 15:46:51.718775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.718722 5476 scope.go:122] "RemoveContainer" containerID="084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8" Nov 02 15:46:51.731693 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.731675 5476 scope.go:122] "RemoveContainer" containerID="084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8" Nov 02 15:46:51.732073 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:46:51.732050 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8\": container with ID starting with 084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8 not found: ID does not exist" containerID="084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8" Nov 02 15:46:51.732139 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.732082 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8"} err="failed to get container status \"084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8\": rpc error: code = NotFound desc = could not find container \"084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8\": container with ID starting with 084eba72c5dc3bc677136a8e7e8145ebd4255ea07662033b19111331ab31f8e8 not found: ID does not exist" Nov 02 15:46:51.872149 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.872114 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg"] Nov 02 15:46:51.923255 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:51.923225 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2203c8-54d447875f-72xsg"] Nov 02 15:46:53.924674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:46:53.924628 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd84d132-0deb-4c37-943d-1bb1a411fb41" path="/var/lib/kubelet/pods/bd84d132-0deb-4c37-943d-1bb1a411fb41/volumes" Nov 02 15:47:02.069178 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:02.069139 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm"] Nov 02 15:47:17.089895 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.089851 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" containerID="cri-o://b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042" gracePeriod=45 Nov 02 15:47:17.355088 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.355064 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:47:17.418209 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.418176 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5ffe407-4378-44c8-846d-fafc02f5d26f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ffe407-4378-44c8-846d-fafc02f5d26f-tls-certs\") pod \"f5ffe407-4378-44c8-846d-fafc02f5d26f\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " Nov 02 15:47:17.418408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.418230 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-model-cache\") pod \"f5ffe407-4378-44c8-846d-fafc02f5d26f\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " Nov 02 15:47:17.418408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.418251 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-tmp-dir\") pod \"f5ffe407-4378-44c8-846d-fafc02f5d26f\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " Nov 02 15:47:17.418408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.418271 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ffe407-4378-44c8-846d-fafc02f5d26f-kube-api-access-hsrqj\" (UniqueName: \"kubernetes.io/projected/f5ffe407-4378-44c8-846d-fafc02f5d26f-kube-api-access-hsrqj\") pod \"f5ffe407-4378-44c8-846d-fafc02f5d26f\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " Nov 02 15:47:17.418408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.418334 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-home\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-home\") pod \"f5ffe407-4378-44c8-846d-fafc02f5d26f\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " Nov 02 15:47:17.418651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.418438 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-dshm\") pod \"f5ffe407-4378-44c8-846d-fafc02f5d26f\" (UID: \"f5ffe407-4378-44c8-846d-fafc02f5d26f\") " Nov 02 15:47:17.418651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.418589 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-tmp-dir" pod "f5ffe407-4378-44c8-846d-fafc02f5d26f" (UID: "f5ffe407-4378-44c8-846d-fafc02f5d26f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.418651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.418577 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-model-cache" pod "f5ffe407-4378-44c8-846d-fafc02f5d26f" (UID: "f5ffe407-4378-44c8-846d-fafc02f5d26f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.418796 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.418655 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-home" pod "f5ffe407-4378-44c8-846d-fafc02f5d26f" (UID: "f5ffe407-4378-44c8-846d-fafc02f5d26f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.419074 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.419050 5476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-model-cache\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.419151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.419080 5476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-tmp-dir\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.419151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.419096 5476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-home\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.420645 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.420617 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-dshm" pod "f5ffe407-4378-44c8-846d-fafc02f5d26f" (UID: "f5ffe407-4378-44c8-846d-fafc02f5d26f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.420772 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.420747 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ffe407-4378-44c8-846d-fafc02f5d26f-tls-certs" pod "f5ffe407-4378-44c8-846d-fafc02f5d26f" (UID: "f5ffe407-4378-44c8-846d-fafc02f5d26f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:17.420818 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.420779 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ffe407-4378-44c8-846d-fafc02f5d26f-kube-api-access-hsrqj" pod "f5ffe407-4378-44c8-846d-fafc02f5d26f" (UID: "f5ffe407-4378-44c8-846d-fafc02f5d26f"). InnerVolumeSpecName "kube-api-access-hsrqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:17.520092 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.520049 5476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5ffe407-4378-44c8-846d-fafc02f5d26f-tls-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.520092 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.520094 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsrqj\" (UniqueName: \"kubernetes.io/projected/f5ffe407-4378-44c8-846d-fafc02f5d26f-kube-api-access-hsrqj\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.520350 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.520110 5476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5ffe407-4378-44c8-846d-fafc02f5d26f-dshm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.839537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.839408 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:47:17.889873 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.889840 5476 generic.go:356] "Generic (PLEG): container finished" podID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerID="b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042" exitCode=0 Nov 02 15:47:17.890085 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.889898 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" Nov 02 15:47:17.890085 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.889918 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" event={"ID":"f5ffe407-4378-44c8-846d-fafc02f5d26f","Type":"ContainerDied","Data":"b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042"} Nov 02 15:47:17.890085 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.889964 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm" event={"ID":"f5ffe407-4378-44c8-846d-fafc02f5d26f","Type":"ContainerDied","Data":"568666776a8a49f040212e9cebaef583b50cc8cdb68569f06578fb16581990c1"} Nov 02 15:47:17.890085 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.889980 5476 scope.go:122] "RemoveContainer" containerID="b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042" Nov 02 15:47:17.900153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.900133 5476 scope.go:122] "RemoveContainer" containerID="b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042" Nov 02 15:47:17.900473 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:47:17.900441 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042\": container with ID starting with b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042 not found: ID does not exist" containerID="b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042" Nov 02 15:47:17.900473 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:17.900471 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042"} err="failed to get container status \"b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042\": rpc error: code = NotFound desc = could not find container \"b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042\": container with ID starting with b53f81ef069e69170ff9ef42278be2d33f7a7c49363086cb4d35c6435378f042 not found: ID does not exist" Nov 02 15:47:18.051825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:18.051788 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm"] Nov 02 15:47:18.054477 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:18.054447 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/simulator-w6-e98976-kserve-8645fc65b5-sc2pm"] Nov 02 15:47:19.633594 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:19.633559 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq"] Nov 02 15:47:19.634150 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:19.633848 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" containerID="cri-o://7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0" gracePeriod=30 Nov 02 15:47:19.922733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:19.922644 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" path="/var/lib/kubelet/pods/f5ffe407-4378-44c8-846d-fafc02f5d26f/volumes" Nov 02 15:47:20.148755 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:47:20.148714 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:47:24.904156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:24.904078 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:47:24.936269 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:24.936231 5476 generic.go:356] "Generic (PLEG): container finished" podID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerID="7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0" exitCode=0 Nov 02 15:47:24.936464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:24.936284 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" Nov 02 15:47:24.936464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:24.936312 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" event={"ID":"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5","Type":"ContainerDied","Data":"7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0"} Nov 02 15:47:24.936464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:24.936388 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq" event={"ID":"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5","Type":"ContainerDied","Data":"8349424c9aa7ed66e023e2094c7352f3d9fc19ac47e7a8c665c7f36e53b95eaa"} Nov 02 15:47:24.936464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:24.936413 5476 scope.go:122] "RemoveContainer" containerID="7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0" Nov 02 15:47:24.946487 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:24.946465 5476 scope.go:122] "RemoveContainer" containerID="fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8" Nov 02 15:47:24.956117 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:24.956094 5476 scope.go:122] "RemoveContainer" containerID="7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0" Nov 02 15:47:24.956456 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:47:24.956432 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0\": container with ID starting with 7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0 not found: ID does not exist" containerID="7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0" Nov 02 15:47:24.956514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:24.956466 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0"} err="failed to get container status \"7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0\": rpc error: code = NotFound desc = could not find container \"7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0\": container with ID starting with 7da0c2479fdfb122d01e3febd1b26ffd635d83a9a460b41eba66449f24a458e0 not found: ID does not exist" Nov 02 15:47:24.956514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:24.956485 5476 scope.go:122] "RemoveContainer" containerID="fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8" Nov 02 15:47:24.956757 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:47:24.956738 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8\": container with ID starting with fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8 not found: ID does not exist" containerID="fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8" Nov 02 15:47:24.956825 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:24.956770 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8"} err="failed to get container status \"fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8\": rpc error: code = NotFound desc = could not find container \"fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8\": container with ID starting with fb0c2ac7099142699996abbd4e9a3cf55aa81a7176b90f3893decc8a3000daa8 not found: ID does not exist" Nov 02 15:47:25.004667 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.004627 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-data\") pod \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " Nov 02 15:47:25.004861 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.004694 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-credential-socket\") pod \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " Nov 02 15:47:25.004861 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.004788 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-envoy\") pod \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " Nov 02 15:47:25.005035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.004872 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-socket\") pod \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " Nov 02 15:47:25.005035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.004906 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-certs\") pod \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " Nov 02 15:47:25.005035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.004948 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-podinfo\") pod \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " Nov 02 15:47:25.005035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.004925 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-data" pod "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" (UID: "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:25.005035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.004978 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istiod-ca-cert\") pod \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " Nov 02 15:47:25.005323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.005037 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-credential-socket" pod "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" (UID: "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:25.005323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.005051 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-token\" (UniqueName: \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-token\") pod \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " Nov 02 15:47:25.005323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.005144 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-kube-api-access-7nf86\" (UniqueName: \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-kube-api-access-7nf86\") pod \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\" (UID: \"df47ed4a-e6d2-4c9d-b29a-61830fb32bf5\") " Nov 02 15:47:25.005323 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.005225 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-socket" pod "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" (UID: "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:25.005575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.005370 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istiod-ca-cert" pod "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" (UID: "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:25.005575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.005528 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-certs" pod "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" (UID: "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:25.005673 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.005655 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:25.005720 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.005680 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:25.005720 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.005696 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:25.005720 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.005710 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:25.005874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.005723 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:25.007286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.007246 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-envoy" pod "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" (UID: "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:25.007730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.007701 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-kube-api-access-7nf86" pod "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" (UID: "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5"). InnerVolumeSpecName "kube-api-access-7nf86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:25.007730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.007711 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-token" pod "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" (UID: "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:25.007730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.007716 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-podinfo" pod "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" (UID: "df47ed4a-e6d2-4c9d-b29a-61830fb32bf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:47:25.106711 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.106613 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:25.106711 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.106651 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:25.106711 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.106668 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:25.106711 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.106681 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nf86\" (UniqueName: \"kubernetes.io/projected/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5-kube-api-access-7nf86\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:25.351358 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.351322 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq"] Nov 02 15:47:25.361194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.361151 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2203c8-openshift-default-5bcd55df55-z8kgq"] Nov 02 15:47:25.922678 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:25.922640 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" path="/var/lib/kubelet/pods/df47ed4a-e6d2-4c9d-b29a-61830fb32bf5/volumes" Nov 02 15:47:31.857866 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:31.857829 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2"] Nov 02 15:47:31.904075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:31.904035 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v"] Nov 02 15:47:31.906484 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:31.906447 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq"] Nov 02 15:47:38.577748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:38.577715 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7"] Nov 02 15:47:38.624356 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:38.624295 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz"] Nov 02 15:47:43.098341 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:43.098307 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:47:45.634096 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:45.634061 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl"] Nov 02 15:47:45.812324 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:45.812288 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt"] Nov 02 15:47:46.879452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:46.879408 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" containerID="cri-o://94f16287d6e93e58544e301675d1c07769ba4731f8a7487de92ef7c31e6af05c" gracePeriod=45 Nov 02 15:47:46.926715 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:46.926659 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" containerID="cri-o://6f2b8afc1145f5b43b5a0ab82d55360a71d9f9ce93380a6ebfffa420b96974b8" gracePeriod=45 Nov 02 15:47:46.927482 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:46.927434 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" containerID="cri-o://82fd49ee96788f5bc900ef774449e3546177e7501e0fc9872563f0c01e43ed3e" gracePeriod=45 Nov 02 15:47:47.083709 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.083661 5476 generic.go:356] "Generic (PLEG): container finished" podID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerID="6f2b8afc1145f5b43b5a0ab82d55360a71d9f9ce93380a6ebfffa420b96974b8" exitCode=0 Nov 02 15:47:47.083880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.083775 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" event={"ID":"00be7b9f-4f51-4c7e-b795-76e23f06adeb","Type":"ContainerDied","Data":"6f2b8afc1145f5b43b5a0ab82d55360a71d9f9ce93380a6ebfffa420b96974b8"} Nov 02 15:47:47.085796 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.085769 5476 generic.go:356] "Generic (PLEG): container finished" podID="aec73889-564d-462b-9b8b-fa969f995df7" containerID="94f16287d6e93e58544e301675d1c07769ba4731f8a7487de92ef7c31e6af05c" exitCode=0 Nov 02 15:47:47.085952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.085932 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" event={"ID":"aec73889-564d-462b-9b8b-fa969f995df7","Type":"ContainerDied","Data":"94f16287d6e93e58544e301675d1c07769ba4731f8a7487de92ef7c31e6af05c"} Nov 02 15:47:47.087461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.087436 5476 generic.go:356] "Generic (PLEG): container finished" podID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerID="82fd49ee96788f5bc900ef774449e3546177e7501e0fc9872563f0c01e43ed3e" exitCode=0 Nov 02 15:47:47.087580 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.087523 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" event={"ID":"6c629e7a-d8a7-4073-ac68-4193ec218328","Type":"ContainerDied","Data":"82fd49ee96788f5bc900ef774449e3546177e7501e0fc9872563f0c01e43ed3e"} Nov 02 15:47:47.213126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.213090 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:47:47.255520 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.255487 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:47:47.257559 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.257531 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec73889-564d-462b-9b8b-fa969f995df7-kube-api-access-djt9z\" (UniqueName: \"kubernetes.io/projected/aec73889-564d-462b-9b8b-fa969f995df7-kube-api-access-djt9z\") pod \"aec73889-564d-462b-9b8b-fa969f995df7\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " Nov 02 15:47:47.257707 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.257639 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-dshm\") pod \"aec73889-564d-462b-9b8b-fa969f995df7\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " Nov 02 15:47:47.257707 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.257668 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-tmp-dir\") pod \"aec73889-564d-462b-9b8b-fa969f995df7\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " Nov 02 15:47:47.257833 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.257724 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-home\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-home\") pod \"aec73889-564d-462b-9b8b-fa969f995df7\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " Nov 02 15:47:47.257833 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.257757 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aec73889-564d-462b-9b8b-fa969f995df7-tls-certs\" (UniqueName: \"kubernetes.io/secret/aec73889-564d-462b-9b8b-fa969f995df7-tls-certs\") pod \"aec73889-564d-462b-9b8b-fa969f995df7\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " Nov 02 15:47:47.258130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.258078 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-home" pod "aec73889-564d-462b-9b8b-fa969f995df7" (UID: "aec73889-564d-462b-9b8b-fa969f995df7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.258130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.258092 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-tmp-dir" pod "aec73889-564d-462b-9b8b-fa969f995df7" (UID: "aec73889-564d-462b-9b8b-fa969f995df7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.258446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.258406 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-model-cache" pod "aec73889-564d-462b-9b8b-fa969f995df7" (UID: "aec73889-564d-462b-9b8b-fa969f995df7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.258590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.257793 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-model-cache\") pod \"aec73889-564d-462b-9b8b-fa969f995df7\" (UID: \"aec73889-564d-462b-9b8b-fa969f995df7\") " Nov 02 15:47:47.259086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.259068 5476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-tmp-dir\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.259086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.259087 5476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-home\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.259249 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.259095 5476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-model-cache\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.260254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.260220 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-dshm" pod "aec73889-564d-462b-9b8b-fa969f995df7" (UID: "aec73889-564d-462b-9b8b-fa969f995df7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.260349 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.260333 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec73889-564d-462b-9b8b-fa969f995df7-kube-api-access-djt9z" pod "aec73889-564d-462b-9b8b-fa969f995df7" (UID: "aec73889-564d-462b-9b8b-fa969f995df7"). InnerVolumeSpecName "kube-api-access-djt9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:47.260411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.260380 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aec73889-564d-462b-9b8b-fa969f995df7-tls-certs" pod "aec73889-564d-462b-9b8b-fa969f995df7" (UID: "aec73889-564d-462b-9b8b-fa969f995df7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:47.268400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.268377 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:47:47.360573 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.360494 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00be7b9f-4f51-4c7e-b795-76e23f06adeb-kube-api-access-2qsvc\" (UniqueName: \"kubernetes.io/projected/00be7b9f-4f51-4c7e-b795-76e23f06adeb-kube-api-access-2qsvc\") pod \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " Nov 02 15:47:47.360771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.360623 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-home\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-home\") pod \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " Nov 02 15:47:47.360771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.360660 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tls-certs\" (UniqueName: \"kubernetes.io/secret/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tls-certs\") pod \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " Nov 02 15:47:47.360771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.360694 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-model-cache\") pod \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " Nov 02 15:47:47.360771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.360762 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-dshm\") pod \"6c629e7a-d8a7-4073-ac68-4193ec218328\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " Nov 02 15:47:47.361033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.360795 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-tmp-dir\") pod \"6c629e7a-d8a7-4073-ac68-4193ec218328\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " Nov 02 15:47:47.361033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.360858 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c629e7a-d8a7-4073-ac68-4193ec218328-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c629e7a-d8a7-4073-ac68-4193ec218328-tls-certs\") pod \"6c629e7a-d8a7-4073-ac68-4193ec218328\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " Nov 02 15:47:47.361033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.360937 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c629e7a-d8a7-4073-ac68-4193ec218328-kube-api-access-jdk6s\" (UniqueName: \"kubernetes.io/projected/6c629e7a-d8a7-4073-ac68-4193ec218328-kube-api-access-jdk6s\") pod \"6c629e7a-d8a7-4073-ac68-4193ec218328\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " Nov 02 15:47:47.361210 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.361109 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tmp-dir\") pod \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " Nov 02 15:47:47.361210 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.361142 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-dshm\") pod \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\" (UID: \"00be7b9f-4f51-4c7e-b795-76e23f06adeb\") " Nov 02 15:47:47.361210 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.361177 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-home\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-home\") pod \"6c629e7a-d8a7-4073-ac68-4193ec218328\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " Nov 02 15:47:47.361210 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.361204 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-model-cache\") pod \"6c629e7a-d8a7-4073-ac68-4193ec218328\" (UID: \"6c629e7a-d8a7-4073-ac68-4193ec218328\") " Nov 02 15:47:47.362326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.361927 5476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aec73889-564d-462b-9b8b-fa969f995df7-dshm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.362326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.361989 5476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aec73889-564d-462b-9b8b-fa969f995df7-tls-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.362326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.362022 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djt9z\" (UniqueName: \"kubernetes.io/projected/aec73889-564d-462b-9b8b-fa969f995df7-kube-api-access-djt9z\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.362326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.360933 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-home" pod "00be7b9f-4f51-4c7e-b795-76e23f06adeb" (UID: "00be7b9f-4f51-4c7e-b795-76e23f06adeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.362326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.361125 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-model-cache" pod "00be7b9f-4f51-4c7e-b795-76e23f06adeb" (UID: "00be7b9f-4f51-4c7e-b795-76e23f06adeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.362326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.361783 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-tmp-dir" pod "6c629e7a-d8a7-4073-ac68-4193ec218328" (UID: "6c629e7a-d8a7-4073-ac68-4193ec218328"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.362326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.361961 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-home" pod "6c629e7a-d8a7-4073-ac68-4193ec218328" (UID: "6c629e7a-d8a7-4073-ac68-4193ec218328"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.362326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.362249 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-model-cache" pod "6c629e7a-d8a7-4073-ac68-4193ec218328" (UID: "6c629e7a-d8a7-4073-ac68-4193ec218328"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.362813 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.362465 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tmp-dir" pod "00be7b9f-4f51-4c7e-b795-76e23f06adeb" (UID: "00be7b9f-4f51-4c7e-b795-76e23f06adeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.363508 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.363465 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00be7b9f-4f51-4c7e-b795-76e23f06adeb-kube-api-access-2qsvc" pod "00be7b9f-4f51-4c7e-b795-76e23f06adeb" (UID: "00be7b9f-4f51-4c7e-b795-76e23f06adeb"). InnerVolumeSpecName "kube-api-access-2qsvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:47.363733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.363705 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tls-certs" pod "00be7b9f-4f51-4c7e-b795-76e23f06adeb" (UID: "00be7b9f-4f51-4c7e-b795-76e23f06adeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:47.364753 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.364729 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c629e7a-d8a7-4073-ac68-4193ec218328-tls-certs" pod "6c629e7a-d8a7-4073-ac68-4193ec218328" (UID: "6c629e7a-d8a7-4073-ac68-4193ec218328"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:47.364879 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.364755 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-dshm" pod "00be7b9f-4f51-4c7e-b795-76e23f06adeb" (UID: "00be7b9f-4f51-4c7e-b795-76e23f06adeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.365149 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.365122 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-dshm" pod "6c629e7a-d8a7-4073-ac68-4193ec218328" (UID: "6c629e7a-d8a7-4073-ac68-4193ec218328"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.365421 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.365398 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c629e7a-d8a7-4073-ac68-4193ec218328-kube-api-access-jdk6s" pod "6c629e7a-d8a7-4073-ac68-4193ec218328" (UID: "6c629e7a-d8a7-4073-ac68-4193ec218328"). InnerVolumeSpecName "kube-api-access-jdk6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:47.463248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463154 5476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-tmp-dir\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.463248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463197 5476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c629e7a-d8a7-4073-ac68-4193ec218328-tls-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.463248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463208 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdk6s\" (UniqueName: \"kubernetes.io/projected/6c629e7a-d8a7-4073-ac68-4193ec218328-kube-api-access-jdk6s\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.463248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463218 5476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tmp-dir\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.463248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463228 5476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-dshm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.463248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463236 5476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-home\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.463248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463244 5476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-model-cache\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.463748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463254 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qsvc\" (UniqueName: \"kubernetes.io/projected/00be7b9f-4f51-4c7e-b795-76e23f06adeb-kube-api-access-2qsvc\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.463748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463267 5476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-home\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.463748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463279 5476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00be7b9f-4f51-4c7e-b795-76e23f06adeb-tls-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.463748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463286 5476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00be7b9f-4f51-4c7e-b795-76e23f06adeb-model-cache\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.463748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:47.463294 5476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c629e7a-d8a7-4073-ac68-4193ec218328-dshm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:48.094593 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.094566 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" Nov 02 15:47:48.095057 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.094601 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2" event={"ID":"aec73889-564d-462b-9b8b-fa969f995df7","Type":"ContainerDied","Data":"f1f0471beea00c48d364d52b40e39828e0b9a6f853a2daf7d38bad40e7323ebe"} Nov 02 15:47:48.095057 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.094643 5476 scope.go:122] "RemoveContainer" containerID="94f16287d6e93e58544e301675d1c07769ba4731f8a7487de92ef7c31e6af05c" Nov 02 15:47:48.096623 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.096600 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" Nov 02 15:47:48.097059 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.096604 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq" event={"ID":"6c629e7a-d8a7-4073-ac68-4193ec218328","Type":"ContainerDied","Data":"5be193802e5e979362069e52ca3a47302aeb3bac30aba138f879b50f1bfe6250"} Nov 02 15:47:48.098274 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.098253 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" event={"ID":"00be7b9f-4f51-4c7e-b795-76e23f06adeb","Type":"ContainerDied","Data":"aaf1e5e368eee45b40c92787d413f0167f96f001438e8517cf0835fe5c7e4a7d"} Nov 02 15:47:48.098371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.098294 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v" Nov 02 15:47:48.105554 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.105537 5476 scope.go:122] "RemoveContainer" containerID="82fd49ee96788f5bc900ef774449e3546177e7501e0fc9872563f0c01e43ed3e" Nov 02 15:47:48.114101 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.114071 5476 scope.go:122] "RemoveContainer" containerID="6f2b8afc1145f5b43b5a0ab82d55360a71d9f9ce93380a6ebfffa420b96974b8" Nov 02 15:47:48.250518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.250428 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq"] Nov 02 15:47:48.296465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.296432 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-624908/simulator-w1-624908-kserve-7cc675b85-2ggnq"] Nov 02 15:47:48.358395 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.358358 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2"] Nov 02 15:47:48.362357 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.362327 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-w1-624908-kserve-6c84d5f55f-w2bq2"] Nov 02 15:47:48.464925 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.464881 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v"] Nov 02 15:47:48.471112 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:48.471083 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-624908/premium-w1-624908-kserve-679ccd8856-l522v"] Nov 02 15:47:49.921474 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:49.921432 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" path="/var/lib/kubelet/pods/00be7b9f-4f51-4c7e-b795-76e23f06adeb/volumes" Nov 02 15:47:49.922220 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:49.922198 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" path="/var/lib/kubelet/pods/6c629e7a-d8a7-4073-ac68-4193ec218328/volumes" Nov 02 15:47:49.922641 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:49.922622 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec73889-564d-462b-9b8b-fa969f995df7" path="/var/lib/kubelet/pods/aec73889-564d-462b-9b8b-fa969f995df7/volumes" Nov 02 15:47:50.146615 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:47:50.146581 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:47:50.634662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:50.634610 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" podUID="f3570096-5eee-4999-ae44-e3386b5e96f3" containerName="payload-processing" containerID="cri-o://4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270" gracePeriod=25 Nov 02 15:47:50.813416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:50.813365 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" podUID="d945fed9-7518-4fd5-bb42-ef6d721c1aba" containerName="payload-pre-processing" containerID="cri-o://913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670" gracePeriod=25 Nov 02 15:47:51.646161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:51.646120 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" podUID="f3570096-5eee-4999-ae44-e3386b5e96f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:47:51.646161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:51.646136 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" podUID="d945fed9-7518-4fd5-bb42-ef6d721c1aba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:47:52.183130 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.183094 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh"] Nov 02 15:47:52.183333 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.183311 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" podUID="7b128868-d225-48c1-bf70-b6d3d9d40ffc" containerName="maas-api" containerID="cri-o://e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750" gracePeriod=30 Nov 02 15:47:52.456749 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.456713 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:47:52.518189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.518158 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-metrics-certs\") pod \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " Nov 02 15:47:52.518367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.518252 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b128868-d225-48c1-bf70-b6d3d9d40ffc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b128868-d225-48c1-bf70-b6d3d9d40ffc-openshift-service-ca\") pod \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " Nov 02 15:47:52.518367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.518285 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-maas-api-tls\") pod \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " Nov 02 15:47:52.518367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.518345 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b128868-d225-48c1-bf70-b6d3d9d40ffc-kube-api-access-9b2c6\" (UniqueName: \"kubernetes.io/projected/7b128868-d225-48c1-bf70-b6d3d9d40ffc-kube-api-access-9b2c6\") pod \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\" (UID: \"7b128868-d225-48c1-bf70-b6d3d9d40ffc\") " Nov 02 15:47:52.518685 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.518645 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b128868-d225-48c1-bf70-b6d3d9d40ffc-openshift-service-ca" pod "7b128868-d225-48c1-bf70-b6d3d9d40ffc" (UID: "7b128868-d225-48c1-bf70-b6d3d9d40ffc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:52.520381 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.520356 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-metrics-certs" pod "7b128868-d225-48c1-bf70-b6d3d9d40ffc" (UID: "7b128868-d225-48c1-bf70-b6d3d9d40ffc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:52.520486 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.520377 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-maas-api-tls" pod "7b128868-d225-48c1-bf70-b6d3d9d40ffc" (UID: "7b128868-d225-48c1-bf70-b6d3d9d40ffc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:52.520486 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.520446 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b128868-d225-48c1-bf70-b6d3d9d40ffc-kube-api-access-9b2c6" pod "7b128868-d225-48c1-bf70-b6d3d9d40ffc" (UID: "7b128868-d225-48c1-bf70-b6d3d9d40ffc"). InnerVolumeSpecName "kube-api-access-9b2c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:52.619822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.619781 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:52.619822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.619815 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b128868-d225-48c1-bf70-b6d3d9d40ffc-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:52.619822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.619825 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b128868-d225-48c1-bf70-b6d3d9d40ffc-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:52.620167 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:52.619835 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b2c6\" (UniqueName: \"kubernetes.io/projected/7b128868-d225-48c1-bf70-b6d3d9d40ffc-kube-api-access-9b2c6\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:53.166573 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.166534 5476 generic.go:356] "Generic (PLEG): container finished" podID="7b128868-d225-48c1-bf70-b6d3d9d40ffc" containerID="e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750" exitCode=0 Nov 02 15:47:53.167134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.166621 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" event={"ID":"7b128868-d225-48c1-bf70-b6d3d9d40ffc","Type":"ContainerDied","Data":"e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750"} Nov 02 15:47:53.167134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.166658 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" Nov 02 15:47:53.167134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.166685 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh" event={"ID":"7b128868-d225-48c1-bf70-b6d3d9d40ffc","Type":"ContainerDied","Data":"954ba15ab34a2ac1d76053be4e34f5933b76c03dd2971ec04e75271a33b12784"} Nov 02 15:47:53.167134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.166711 5476 scope.go:122] "RemoveContainer" containerID="e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750" Nov 02 15:47:53.178034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.178013 5476 scope.go:122] "RemoveContainer" containerID="e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750" Nov 02 15:47:53.178445 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:47:53.178416 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750\": container with ID starting with e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750 not found: ID does not exist" containerID="e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750" Nov 02 15:47:53.178545 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.178454 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750"} err="failed to get container status \"e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750\": rpc error: code = NotFound desc = could not find container \"e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750\": container with ID starting with e317b60f575166e0f4e78755f726ee7408e9216ca807cab00693f12cef208750 not found: ID does not exist" Nov 02 15:47:53.321414 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.321382 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh"] Nov 02 15:47:53.367776 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.367744 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f6b245-ff8ff4748-n9xmh"] Nov 02 15:47:53.600418 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.600324 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" containerID="cri-o://018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c" gracePeriod=45 Nov 02 15:47:53.640646 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:47:53.640612 5476 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecf2b48f_db4b_4720_91e7_d0eaabd1d76c.slice/crio-018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecf2b48f_db4b_4720_91e7_d0eaabd1d76c.slice/crio-conmon-018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:53.651653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.651615 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" containerID="cri-o://13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb" gracePeriod=45 Nov 02 15:47:53.719481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.719444 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": dial tcp 10.134.0.84:8000: connect: connection refused" Nov 02 15:47:53.918583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.918555 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:47:53.922246 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.922215 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b128868-d225-48c1-bf70-b6d3d9d40ffc" path="/var/lib/kubelet/pods/7b128868-d225-48c1-bf70-b6d3d9d40ffc/volumes" Nov 02 15:47:53.950417 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:53.950389 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:47:54.032461 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032428 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-home\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-home\") pod \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " Nov 02 15:47:54.032647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032481 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-model-cache\") pod \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " Nov 02 15:47:54.032647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032528 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-kube-api-access-dv2bc\" (UniqueName: \"kubernetes.io/projected/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-kube-api-access-dv2bc\") pod \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " Nov 02 15:47:54.032647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032577 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tmp-dir\") pod \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " Nov 02 15:47:54.032835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032646 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-kube-api-access-8kkqc\" (UniqueName: \"kubernetes.io/projected/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-kube-api-access-8kkqc\") pod \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " Nov 02 15:47:54.032835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032673 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tls-certs\") pod \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " Nov 02 15:47:54.032835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032706 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tls-certs\") pod \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " Nov 02 15:47:54.032835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032735 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-home" pod "7dba751d-d54a-46a8-96eb-5adb4cbc66ef" (UID: "7dba751d-d54a-46a8-96eb-5adb4cbc66ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.032835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032768 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-home\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-home\") pod \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " Nov 02 15:47:54.032835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032772 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-model-cache" pod "7dba751d-d54a-46a8-96eb-5adb4cbc66ef" (UID: "7dba751d-d54a-46a8-96eb-5adb4cbc66ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.032835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032831 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-dshm\") pod \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\" (UID: \"7dba751d-d54a-46a8-96eb-5adb4cbc66ef\") " Nov 02 15:47:54.033220 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032861 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-model-cache\") pod \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " Nov 02 15:47:54.033220 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032944 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tmp-dir\") pod \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " Nov 02 15:47:54.033220 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.032975 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-dshm\") pod \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\" (UID: \"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c\") " Nov 02 15:47:54.033414 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.033398 5476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-home\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.033480 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.033419 5476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-model-cache\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.036651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.035482 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tls-certs" pod "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" (UID: "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:54.036651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.035555 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tls-certs" pod "7dba751d-d54a-46a8-96eb-5adb4cbc66ef" (UID: "7dba751d-d54a-46a8-96eb-5adb4cbc66ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:54.036651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.035692 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-model-cache" pod "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" (UID: "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.036651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.035786 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tmp-dir" pod "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" (UID: "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.036651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.035807 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-dshm" pod "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" (UID: "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.036651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.035863 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-home" pod "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" (UID: "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.036651 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.035902 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tmp-dir" pod "7dba751d-d54a-46a8-96eb-5adb4cbc66ef" (UID: "7dba751d-d54a-46a8-96eb-5adb4cbc66ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.038385 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.038356 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-kube-api-access-8kkqc" pod "7dba751d-d54a-46a8-96eb-5adb4cbc66ef" (UID: "7dba751d-d54a-46a8-96eb-5adb4cbc66ef"). InnerVolumeSpecName "kube-api-access-8kkqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:54.038510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.038433 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-dshm" pod "7dba751d-d54a-46a8-96eb-5adb4cbc66ef" (UID: "7dba751d-d54a-46a8-96eb-5adb4cbc66ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.038579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.038510 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-kube-api-access-dv2bc" pod "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" (UID: "ecf2b48f-db4b-4720-91e7-d0eaabd1d76c"). InnerVolumeSpecName "kube-api-access-dv2bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:54.134392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.134301 5476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-home\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.134392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.134333 5476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-dshm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.134392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.134346 5476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-model-cache\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.134392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.134359 5476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tmp-dir\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.134392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.134370 5476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-dshm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.134392 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.134382 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv2bc\" (UniqueName: \"kubernetes.io/projected/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-kube-api-access-dv2bc\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.134706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.134396 5476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tmp-dir\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.134706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.134409 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kkqc\" (UniqueName: \"kubernetes.io/projected/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-kube-api-access-8kkqc\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.134706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.134420 5476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dba751d-d54a-46a8-96eb-5adb4cbc66ef-tls-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.134706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.134435 5476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c-tls-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.184273 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.184230 5476 generic.go:356] "Generic (PLEG): container finished" podID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerID="13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb" exitCode=0 Nov 02 15:47:54.184792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.184416 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" Nov 02 15:47:54.184792 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.184756 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" event={"ID":"7dba751d-d54a-46a8-96eb-5adb4cbc66ef","Type":"ContainerDied","Data":"13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb"} Nov 02 15:47:54.184919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.184810 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" event={"ID":"7dba751d-d54a-46a8-96eb-5adb4cbc66ef","Type":"ContainerDied","Data":"c6799a2de72336902f53c52a3d99c93c396cf41e7f045a68af16e1f084cfecdd"} Nov 02 15:47:54.184919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.184832 5476 scope.go:122] "RemoveContainer" containerID="13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb" Nov 02 15:47:54.187602 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.187573 5476 generic.go:356] "Generic (PLEG): container finished" podID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerID="018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c" exitCode=0 Nov 02 15:47:54.187731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.187623 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" Nov 02 15:47:54.187731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.187645 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" event={"ID":"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c","Type":"ContainerDied","Data":"018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c"} Nov 02 15:47:54.187731 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.187684 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7" event={"ID":"ecf2b48f-db4b-4720-91e7-d0eaabd1d76c","Type":"ContainerDied","Data":"4797cd3355113b082efd04c2c5b6d2dbaeccb3f4fdf9188abd1736834712b68c"} Nov 02 15:47:54.196647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.196625 5476 scope.go:122] "RemoveContainer" containerID="13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb" Nov 02 15:47:54.196966 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:47:54.196940 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb\": container with ID starting with 13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb not found: ID does not exist" containerID="13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb" Nov 02 15:47:54.197035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.196979 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb"} err="failed to get container status \"13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb\": rpc error: code = NotFound desc = could not find container \"13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb\": container with ID starting with 13398566ae38f44fe6b74eb5351046e63a505b5e11ecaf8301cac5a09e81d1cb not found: ID does not exist" Nov 02 15:47:54.197035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.197014 5476 scope.go:122] "RemoveContainer" containerID="018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c" Nov 02 15:47:54.206716 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.206692 5476 scope.go:122] "RemoveContainer" containerID="018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c" Nov 02 15:47:54.207035 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:47:54.207008 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c\": container with ID starting with 018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c not found: ID does not exist" containerID="018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c" Nov 02 15:47:54.207156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.207040 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c"} err="failed to get container status \"018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c\": rpc error: code = NotFound desc = could not find container \"018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c\": container with ID starting with 018fff6159ffb81f05f7ab8767d80a529ea66f43f9724e5bd846f686d8ca3d3c not found: ID does not exist" Nov 02 15:47:54.337566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.337526 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7"] Nov 02 15:47:54.343818 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.343789 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-2-w4-8a2e6f-kserve-6f7898754d-8thr7"] Nov 02 15:47:54.446401 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.446320 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz"] Nov 02 15:47:54.450585 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.450556 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz"] Nov 02 15:47:54.936159 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:54.936116 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/premium-w4-8a2e6f-kserve-c6486b896-l5rxz" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:47:55.922045 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:55.922008 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" path="/var/lib/kubelet/pods/7dba751d-d54a-46a8-96eb-5adb4cbc66ef/volumes" Nov 02 15:47:55.922452 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:47:55.922436 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" path="/var/lib/kubelet/pods/ecf2b48f-db4b-4720-91e7-d0eaabd1d76c/volumes" Nov 02 15:48:01.645868 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:01.645826 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" podUID="f3570096-5eee-4999-ae44-e3386b5e96f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:48:01.646401 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:01.645833 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" podUID="d945fed9-7518-4fd5-bb42-ef6d721c1aba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:48:04.565248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565211 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw"] Nov 02 15:48:04.565724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565643 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b8777e2c-5138-42a2-88ea-2426d7de5ea1" containerName="revoke-keys" Nov 02 15:48:04.565724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565654 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8777e2c-5138-42a2-88ea-2426d7de5ea1" containerName="revoke-keys" Nov 02 15:48:04.565724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565671 5476 cpu_manager.go:414] "containerMap: removing container" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" Nov 02 15:48:04.565724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565676 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" Nov 02 15:48:04.565724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565696 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" Nov 02 15:48:04.565724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565703 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" Nov 02 15:48:04.565724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565716 5476 cpu_manager.go:414] "containerMap: removing container" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerName="cleanup" Nov 02 15:48:04.565724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565721 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerName="cleanup" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565731 5476 cpu_manager.go:414] "containerMap: removing container" podUID="7b128868-d225-48c1-bf70-b6d3d9d40ffc" containerName="maas-api" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565736 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b128868-d225-48c1-bf70-b6d3d9d40ffc" containerName="maas-api" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565744 5476 cpu_manager.go:414] "containerMap: removing container" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565749 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565762 5476 cpu_manager.go:414] "containerMap: removing container" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565769 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565779 5476 cpu_manager.go:414] "containerMap: removing container" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerName="cleanup" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565786 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e92d4a-de8d-4244-912c-9b6d2ddf4b8a" containerName="cleanup" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565795 5476 cpu_manager.go:414] "containerMap: removing container" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565800 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565806 5476 cpu_manager.go:414] "containerMap: removing container" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565810 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565817 5476 cpu_manager.go:414] "containerMap: removing container" podUID="3ff17841-24b5-4837-9a6c-104783665b81" containerName="cleanup" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565822 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff17841-24b5-4837-9a6c-104783665b81" containerName="cleanup" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565829 5476 cpu_manager.go:414] "containerMap: removing container" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565833 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565839 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565843 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565853 5476 cpu_manager.go:414] "containerMap: removing container" podUID="bd84d132-0deb-4c37-943d-1bb1a411fb41" containerName="payload-pre-processing" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565862 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd84d132-0deb-4c37-943d-1bb1a411fb41" containerName="payload-pre-processing" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565872 5476 cpu_manager.go:414] "containerMap: removing container" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565877 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565890 5476 cpu_manager.go:414] "containerMap: removing container" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565895 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565903 5476 cpu_manager.go:414] "containerMap: removing container" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerName="cleanup" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565909 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5c73d7-b426-48b3-bd36-3f628e431f85" containerName="cleanup" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565918 5476 cpu_manager.go:414] "containerMap: removing container" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.565924 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" Nov 02 15:48:04.566033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566036 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ffe407-4378-44c8-846d-fafc02f5d26f" containerName="main" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566050 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b16007-2c3b-4e4c-b08b-4121b3278a65" containerName="cleanup" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566062 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566072 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="00be7b9f-4f51-4c7e-b795-76e23f06adeb" containerName="main" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566081 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566091 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec73889-564d-462b-9b8b-fa969f995df7" containerName="main" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566099 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf2b48f-db4b-4720-91e7-d0eaabd1d76c" containerName="main" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566108 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd84d132-0deb-4c37-943d-1bb1a411fb41" containerName="payload-pre-processing" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566116 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="df47ed4a-e6d2-4c9d-b29a-61830fb32bf5" containerName="istio-proxy" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566126 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d3cf79-d7b4-4d45-8d6a-bdc99a46531b" containerName="istio-proxy" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566136 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eda8baf-c3cb-40ca-9acc-219484ff4dd7" containerName="istio-proxy" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566145 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8777e2c-5138-42a2-88ea-2426d7de5ea1" containerName="revoke-keys" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566155 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dba751d-d54a-46a8-96eb-5adb4cbc66ef" containerName="main" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566163 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c629e7a-d8a7-4073-ac68-4193ec218328" containerName="main" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566171 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6802668-71af-47b7-8cc8-b2c02d9961de" containerName="cleanup" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566181 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b128868-d225-48c1-bf70-b6d3d9d40ffc" containerName="maas-api" Nov 02 15:48:04.567238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.566190 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a8032f-8a4d-412e-a04f-2c479a8a05df" containerName="cleanup" Nov 02 15:48:04.571490 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.571467 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:04.623038 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.622989 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:48:04.657777 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.657743 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948517fb-ptgnw\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:04.657967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.657871 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948517fb-ptgnw\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:04.658058 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.657967 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7m96\" (UniqueName: \"kubernetes.io/projected/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-kube-api-access-g7m96\") pod \"maas-api-revoke-sub-948517fb-ptgnw\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:04.669400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.669368 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:48:04.671787 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.671759 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw"] Nov 02 15:48:04.759775 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.759736 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948517fb-ptgnw\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:04.759956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.759806 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948517fb-ptgnw\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:04.760029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.759956 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7m96\" (UniqueName: \"kubernetes.io/projected/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-kube-api-access-g7m96\") pod \"maas-api-revoke-sub-948517fb-ptgnw\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:04.760523 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.760498 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948517fb-ptgnw\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:04.762263 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.762235 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948517fb-ptgnw\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:04.818237 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.818162 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7m96\" (UniqueName: \"kubernetes.io/projected/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-kube-api-access-g7m96\") pod \"maas-api-revoke-sub-948517fb-ptgnw\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:04.885140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:04.885105 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:05.268333 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:05.268301 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw"] Nov 02 15:48:05.274225 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:48:05.274194 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1b0e7f_4bca_4be2_8f01_02ce88b2b4c1.slice/crio-2fb92134f83371a69455c39f940d0cc00d0b99b0013e00f4115f95492659f702 WatchSource:0}: Error finding container 2fb92134f83371a69455c39f940d0cc00d0b99b0013e00f4115f95492659f702: Status 404 returned error can't find the container with id 2fb92134f83371a69455c39f940d0cc00d0b99b0013e00f4115f95492659f702 Nov 02 15:48:06.264858 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:06.264823 5476 generic.go:356] "Generic (PLEG): container finished" podID="ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1" containerID="ae2653e19aadfdd29660395fc9cf59e482039f0c197ebb72f9ab431afa0b23fd" exitCode=0 Nov 02 15:48:06.264858 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:06.264855 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" event={"ID":"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1","Type":"ContainerDied","Data":"ae2653e19aadfdd29660395fc9cf59e482039f0c197ebb72f9ab431afa0b23fd"} Nov 02 15:48:06.265383 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:06.264902 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" event={"ID":"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1","Type":"ContainerStarted","Data":"2fb92134f83371a69455c39f940d0cc00d0b99b0013e00f4115f95492659f702"} Nov 02 15:48:07.427490 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:07.427437 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:07.595335 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:07.595247 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-kube-api-access-g7m96\" (UniqueName: \"kubernetes.io/projected/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-kube-api-access-g7m96\") pod \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " Nov 02 15:48:07.595557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:07.595392 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-openshift-ca\") pod \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " Nov 02 15:48:07.595557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:07.595451 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-xks-ca\") pod \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\" (UID: \"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1\") " Nov 02 15:48:07.595853 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:07.595827 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-openshift-ca" pod "ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1" (UID: "ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:07.597464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:07.597438 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-kube-api-access-g7m96" pod "ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1" (UID: "ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1"). InnerVolumeSpecName "kube-api-access-g7m96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:07.597566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:07.597544 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-xks-ca" pod "ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1" (UID: "ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:07.697041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:07.696987 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7m96\" (UniqueName: \"kubernetes.io/projected/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-kube-api-access-g7m96\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:07.697041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:07.697039 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:07.697041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:07.697050 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:08.279366 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:08.279281 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" Nov 02 15:48:08.279366 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:08.279308 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw" event={"ID":"ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1","Type":"ContainerDied","Data":"2fb92134f83371a69455c39f940d0cc00d0b99b0013e00f4115f95492659f702"} Nov 02 15:48:08.279366 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:08.279360 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fb92134f83371a69455c39f940d0cc00d0b99b0013e00f4115f95492659f702" Nov 02 15:48:10.941348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:10.941323 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:48:11.032181 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.032151 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3570096-5eee-4999-ae44-e3386b5e96f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3570096-5eee-4999-ae44-e3386b5e96f3-plugins-config-volume\") pod \"f3570096-5eee-4999-ae44-e3386b5e96f3\" (UID: \"f3570096-5eee-4999-ae44-e3386b5e96f3\") " Nov 02 15:48:11.032381 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.032247 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3570096-5eee-4999-ae44-e3386b5e96f3-kube-api-access-8r586\" (UniqueName: \"kubernetes.io/projected/f3570096-5eee-4999-ae44-e3386b5e96f3-kube-api-access-8r586\") pod \"f3570096-5eee-4999-ae44-e3386b5e96f3\" (UID: \"f3570096-5eee-4999-ae44-e3386b5e96f3\") " Nov 02 15:48:11.032625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.032593 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3570096-5eee-4999-ae44-e3386b5e96f3-plugins-config-volume" pod "f3570096-5eee-4999-ae44-e3386b5e96f3" (UID: "f3570096-5eee-4999-ae44-e3386b5e96f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:11.036031 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.035985 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3570096-5eee-4999-ae44-e3386b5e96f3-kube-api-access-8r586" pod "f3570096-5eee-4999-ae44-e3386b5e96f3" (UID: "f3570096-5eee-4999-ae44-e3386b5e96f3"). InnerVolumeSpecName "kube-api-access-8r586". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:11.060697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.060670 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:48:11.133746 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.133708 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3570096-5eee-4999-ae44-e3386b5e96f3-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:11.133746 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.133739 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r586\" (UniqueName: \"kubernetes.io/projected/f3570096-5eee-4999-ae44-e3386b5e96f3-kube-api-access-8r586\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:11.234118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.234028 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d945fed9-7518-4fd5-bb42-ef6d721c1aba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d945fed9-7518-4fd5-bb42-ef6d721c1aba-plugins-config-volume\") pod \"d945fed9-7518-4fd5-bb42-ef6d721c1aba\" (UID: \"d945fed9-7518-4fd5-bb42-ef6d721c1aba\") " Nov 02 15:48:11.234118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.234101 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d945fed9-7518-4fd5-bb42-ef6d721c1aba-kube-api-access-x8zv2\" (UniqueName: \"kubernetes.io/projected/d945fed9-7518-4fd5-bb42-ef6d721c1aba-kube-api-access-x8zv2\") pod \"d945fed9-7518-4fd5-bb42-ef6d721c1aba\" (UID: \"d945fed9-7518-4fd5-bb42-ef6d721c1aba\") " Nov 02 15:48:11.234438 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.234416 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d945fed9-7518-4fd5-bb42-ef6d721c1aba-plugins-config-volume" pod "d945fed9-7518-4fd5-bb42-ef6d721c1aba" (UID: "d945fed9-7518-4fd5-bb42-ef6d721c1aba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:11.234583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.234563 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d945fed9-7518-4fd5-bb42-ef6d721c1aba-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:11.236575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.236546 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d945fed9-7518-4fd5-bb42-ef6d721c1aba-kube-api-access-x8zv2" pod "d945fed9-7518-4fd5-bb42-ef6d721c1aba" (UID: "d945fed9-7518-4fd5-bb42-ef6d721c1aba"). InnerVolumeSpecName "kube-api-access-x8zv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:11.298367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.298332 5476 generic.go:356] "Generic (PLEG): container finished" podID="d945fed9-7518-4fd5-bb42-ef6d721c1aba" containerID="913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670" exitCode=0 Nov 02 15:48:11.298575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.298378 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" Nov 02 15:48:11.298575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.298409 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" event={"ID":"d945fed9-7518-4fd5-bb42-ef6d721c1aba","Type":"ContainerDied","Data":"913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670"} Nov 02 15:48:11.298575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.298465 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt" event={"ID":"d945fed9-7518-4fd5-bb42-ef6d721c1aba","Type":"ContainerDied","Data":"1773bf7bc27c35646929b1331d4a3ecd2db84ed5b6aacc6202b8f0804c517b1d"} Nov 02 15:48:11.298575 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.298493 5476 scope.go:122] "RemoveContainer" containerID="913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670" Nov 02 15:48:11.299899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.299877 5476 generic.go:356] "Generic (PLEG): container finished" podID="f3570096-5eee-4999-ae44-e3386b5e96f3" containerID="4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270" exitCode=0 Nov 02 15:48:11.299980 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.299917 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" Nov 02 15:48:11.300056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.299990 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" event={"ID":"f3570096-5eee-4999-ae44-e3386b5e96f3","Type":"ContainerDied","Data":"4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270"} Nov 02 15:48:11.300056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.300046 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl" event={"ID":"f3570096-5eee-4999-ae44-e3386b5e96f3","Type":"ContainerDied","Data":"7398eec0178bb44cb7a5bece8fbcaeb8d5ffbd1219641f8b22cc427c1cae109f"} Nov 02 15:48:11.309182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.309107 5476 scope.go:122] "RemoveContainer" containerID="913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670" Nov 02 15:48:11.309461 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:48:11.309438 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670\": container with ID starting with 913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670 not found: ID does not exist" containerID="913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670" Nov 02 15:48:11.309565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.309470 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670"} err="failed to get container status \"913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670\": rpc error: code = NotFound desc = could not find container \"913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670\": container with ID starting with 913985afa73ea6b076bebb52a8ee8b3b0255497310d255515f41eab1fd155670 not found: ID does not exist" Nov 02 15:48:11.309565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.309486 5476 scope.go:122] "RemoveContainer" containerID="4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270" Nov 02 15:48:11.318220 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.318204 5476 scope.go:122] "RemoveContainer" containerID="4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270" Nov 02 15:48:11.318495 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:48:11.318475 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270\": container with ID starting with 4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270 not found: ID does not exist" containerID="4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270" Nov 02 15:48:11.318544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.318503 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270"} err="failed to get container status \"4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270\": rpc error: code = NotFound desc = could not find container \"4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270\": container with ID starting with 4b3e82a42ad2c38b48683044a54ac7198987559d8bea4bff072e3710ce863270 not found: ID does not exist" Nov 02 15:48:11.336121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.336098 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8zv2\" (UniqueName: \"kubernetes.io/projected/d945fed9-7518-4fd5-bb42-ef6d721c1aba-kube-api-access-x8zv2\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:11.454313 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.454276 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl"] Nov 02 15:48:11.456582 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.456555 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8a2e6f-569f9455c9-d5vsl"] Nov 02 15:48:11.531203 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.531099 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j"] Nov 02 15:48:11.531399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.531348 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" podUID="eb839896-6a9a-4d11-82a6-597c9d569411" containerName="maas-api" containerID="cri-o://873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691" gracePeriod=30 Nov 02 15:48:11.561694 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.561653 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt"] Nov 02 15:48:11.564177 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.564152 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8a2e6f-5c466d985f-zb2jt"] Nov 02 15:48:11.809663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.809636 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:48:11.935345 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.935300 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d945fed9-7518-4fd5-bb42-ef6d721c1aba" path="/var/lib/kubelet/pods/d945fed9-7518-4fd5-bb42-ef6d721c1aba/volumes" Nov 02 15:48:11.935820 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.935795 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3570096-5eee-4999-ae44-e3386b5e96f3" path="/var/lib/kubelet/pods/f3570096-5eee-4999-ae44-e3386b5e96f3/volumes" Nov 02 15:48:11.951918 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.951878 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb839896-6a9a-4d11-82a6-597c9d569411-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb839896-6a9a-4d11-82a6-597c9d569411-openshift-service-ca\") pod \"eb839896-6a9a-4d11-82a6-597c9d569411\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " Nov 02 15:48:11.952393 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.951926 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-metrics-certs\") pod \"eb839896-6a9a-4d11-82a6-597c9d569411\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " Nov 02 15:48:11.952393 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.952120 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb839896-6a9a-4d11-82a6-597c9d569411-kube-api-access-bgnzh\" (UniqueName: \"kubernetes.io/projected/eb839896-6a9a-4d11-82a6-597c9d569411-kube-api-access-bgnzh\") pod \"eb839896-6a9a-4d11-82a6-597c9d569411\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " Nov 02 15:48:11.952393 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.952231 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb839896-6a9a-4d11-82a6-597c9d569411-openshift-service-ca" pod "eb839896-6a9a-4d11-82a6-597c9d569411" (UID: "eb839896-6a9a-4d11-82a6-597c9d569411"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:11.952393 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.952277 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-maas-api-tls\") pod \"eb839896-6a9a-4d11-82a6-597c9d569411\" (UID: \"eb839896-6a9a-4d11-82a6-597c9d569411\") " Nov 02 15:48:11.952725 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.952702 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb839896-6a9a-4d11-82a6-597c9d569411-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:11.954218 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.954190 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb839896-6a9a-4d11-82a6-597c9d569411-kube-api-access-bgnzh" pod "eb839896-6a9a-4d11-82a6-597c9d569411" (UID: "eb839896-6a9a-4d11-82a6-597c9d569411"). InnerVolumeSpecName "kube-api-access-bgnzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:11.954331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.954272 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-maas-api-tls" pod "eb839896-6a9a-4d11-82a6-597c9d569411" (UID: "eb839896-6a9a-4d11-82a6-597c9d569411"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:11.954386 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:11.954325 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-metrics-certs" pod "eb839896-6a9a-4d11-82a6-597c9d569411" (UID: "eb839896-6a9a-4d11-82a6-597c9d569411"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:12.053668 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.053584 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:12.053668 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.053615 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb839896-6a9a-4d11-82a6-597c9d569411-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:12.053668 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.053625 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgnzh\" (UniqueName: \"kubernetes.io/projected/eb839896-6a9a-4d11-82a6-597c9d569411-kube-api-access-bgnzh\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:12.308926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.308837 5476 generic.go:356] "Generic (PLEG): container finished" podID="eb839896-6a9a-4d11-82a6-597c9d569411" containerID="873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691" exitCode=0 Nov 02 15:48:12.308926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.308875 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" Nov 02 15:48:12.308926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.308899 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" event={"ID":"eb839896-6a9a-4d11-82a6-597c9d569411","Type":"ContainerDied","Data":"873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691"} Nov 02 15:48:12.309253 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.308955 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j" event={"ID":"eb839896-6a9a-4d11-82a6-597c9d569411","Type":"ContainerDied","Data":"3f866d7494a31c0be1ae9811a9ad37b4272ed02f28719653fcdf0234a9153b2e"} Nov 02 15:48:12.309253 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.308986 5476 scope.go:122] "RemoveContainer" containerID="873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691" Nov 02 15:48:12.319821 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.319800 5476 scope.go:122] "RemoveContainer" containerID="873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691" Nov 02 15:48:12.320105 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:48:12.320086 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691\": container with ID starting with 873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691 not found: ID does not exist" containerID="873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691" Nov 02 15:48:12.320175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.320120 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691"} err="failed to get container status \"873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691\": rpc error: code = NotFound desc = could not find container \"873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691\": container with ID starting with 873fe551ffc23b7ba461f7466e2b24a130d101dd06c92205e485251b9e316691 not found: ID does not exist" Nov 02 15:48:12.469411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.469375 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j"] Nov 02 15:48:12.479252 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:12.479212 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-e98976-6c4bf88d99-dcs5j"] Nov 02 15:48:13.922933 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:13.922900 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb839896-6a9a-4d11-82a6-597c9d569411" path="/var/lib/kubelet/pods/eb839896-6a9a-4d11-82a6-597c9d569411/volumes" Nov 02 15:48:14.189830 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:14.189747 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz"] Nov 02 15:48:14.189970 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:14.189954 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerName="istio-proxy" containerID="cri-o://e3f8d5aeda69cfcd9834788bcc0e358745ce5e0f16960bfa0d1e3df7f735fad2" gracePeriod=30 Nov 02 15:48:18.654300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:18.654266 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:48:18.658854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:18.658827 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:48:19.360958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.360916 5476 generic.go:356] "Generic (PLEG): container finished" podID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerID="e3f8d5aeda69cfcd9834788bcc0e358745ce5e0f16960bfa0d1e3df7f735fad2" exitCode=0 Nov 02 15:48:19.361173 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.361152 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" event={"ID":"cc8cd888-ff5b-44a2-b45d-df67e4cf288c","Type":"ContainerDied","Data":"e3f8d5aeda69cfcd9834788bcc0e358745ce5e0f16960bfa0d1e3df7f735fad2"} Nov 02 15:48:19.472017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.471940 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:48:19.537238 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537202 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-socket\") pod \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " Nov 02 15:48:19.537424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537257 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-token\" (UniqueName: \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-token\") pod \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " Nov 02 15:48:19.537424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537304 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-podinfo\") pod \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " Nov 02 15:48:19.537424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537333 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-certs\") pod \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " Nov 02 15:48:19.537424 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537374 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-envoy\") pod \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " Nov 02 15:48:19.537630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537459 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-data\") pod \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " Nov 02 15:48:19.537630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537508 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-socket" pod "cc8cd888-ff5b-44a2-b45d-df67e4cf288c" (UID: "cc8cd888-ff5b-44a2-b45d-df67e4cf288c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:19.537630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537531 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-credential-socket\") pod \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " Nov 02 15:48:19.537630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537578 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-kube-api-access-k4r8k\" (UniqueName: \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-kube-api-access-k4r8k\") pod \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " Nov 02 15:48:19.537630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537613 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istiod-ca-cert\") pod \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\" (UID: \"cc8cd888-ff5b-44a2-b45d-df67e4cf288c\") " Nov 02 15:48:19.537899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537783 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-certs" pod "cc8cd888-ff5b-44a2-b45d-df67e4cf288c" (UID: "cc8cd888-ff5b-44a2-b45d-df67e4cf288c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:19.537899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537769 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-data" pod "cc8cd888-ff5b-44a2-b45d-df67e4cf288c" (UID: "cc8cd888-ff5b-44a2-b45d-df67e4cf288c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:19.537899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.537839 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-credential-socket" pod "cc8cd888-ff5b-44a2-b45d-df67e4cf288c" (UID: "cc8cd888-ff5b-44a2-b45d-df67e4cf288c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:19.538145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.538120 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:19.538145 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.538146 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:19.538294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.538144 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istiod-ca-cert" pod "cc8cd888-ff5b-44a2-b45d-df67e4cf288c" (UID: "cc8cd888-ff5b-44a2-b45d-df67e4cf288c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:19.538294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.538161 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:19.538294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.538175 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:19.539748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.539720 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-podinfo" pod "cc8cd888-ff5b-44a2-b45d-df67e4cf288c" (UID: "cc8cd888-ff5b-44a2-b45d-df67e4cf288c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:48:19.539953 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.539929 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-kube-api-access-k4r8k" pod "cc8cd888-ff5b-44a2-b45d-df67e4cf288c" (UID: "cc8cd888-ff5b-44a2-b45d-df67e4cf288c"). InnerVolumeSpecName "kube-api-access-k4r8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:19.540089 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.540071 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-token" pod "cc8cd888-ff5b-44a2-b45d-df67e4cf288c" (UID: "cc8cd888-ff5b-44a2-b45d-df67e4cf288c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:19.540137 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.540090 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-envoy" pod "cc8cd888-ff5b-44a2-b45d-df67e4cf288c" (UID: "cc8cd888-ff5b-44a2-b45d-df67e4cf288c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:19.639223 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.639184 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:19.639223 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.639217 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:19.639430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.639230 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:19.639430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.639246 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4r8k\" (UniqueName: \"kubernetes.io/projected/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-kube-api-access-k4r8k\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:19.639430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.639261 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8cd888-ff5b-44a2-b45d-df67e4cf288c-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:19.696961 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:19.696928 5476 scope.go:122] "RemoveContainer" containerID="e3f8d5aeda69cfcd9834788bcc0e358745ce5e0f16960bfa0d1e3df7f735fad2" Nov 02 15:48:20.146458 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:48:20.146419 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:48:20.367236 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:20.367191 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" event={"ID":"cc8cd888-ff5b-44a2-b45d-df67e4cf288c","Type":"ContainerDied","Data":"969759fc84a6c74c1d14b0696542510caec8fad6707d86040aaf1a1daefa139c"} Nov 02 15:48:20.367474 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:20.367261 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz" Nov 02 15:48:20.602611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:20.602513 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz"] Nov 02 15:48:20.605812 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:20.603516 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-wthtz"] Nov 02 15:48:21.921286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:21.921251 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" path="/var/lib/kubelet/pods/cc8cd888-ff5b-44a2-b45d-df67e4cf288c/volumes" Nov 02 15:48:25.665882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:25.663858 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:48:27.941674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:27.941634 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g"] Nov 02 15:48:27.942109 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:27.941858 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" containerID="cri-o://11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42" gracePeriod=30 Nov 02 15:48:33.202933 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.202873 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:48:33.285142 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285107 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-kube-api-access-rfv2z\" (UniqueName: \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-kube-api-access-rfv2z\") pod \"be7d2cc8-ea14-45c9-b096-578619c36bf0\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " Nov 02 15:48:33.285142 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285146 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-credential-socket\") pod \"be7d2cc8-ea14-45c9-b096-578619c36bf0\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " Nov 02 15:48:33.285408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285215 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be7d2cc8-ea14-45c9-b096-578619c36bf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7d2cc8-ea14-45c9-b096-578619c36bf0-istiod-ca-cert\") pod \"be7d2cc8-ea14-45c9-b096-578619c36bf0\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " Nov 02 15:48:33.285408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285236 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-socket\") pod \"be7d2cc8-ea14-45c9-b096-578619c36bf0\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " Nov 02 15:48:33.285408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285272 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-token\" (UniqueName: \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-token\") pod \"be7d2cc8-ea14-45c9-b096-578619c36bf0\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " Nov 02 15:48:33.285408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285351 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-podinfo\") pod \"be7d2cc8-ea14-45c9-b096-578619c36bf0\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " Nov 02 15:48:33.285408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285384 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-envoy\") pod \"be7d2cc8-ea14-45c9-b096-578619c36bf0\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " Nov 02 15:48:33.285704 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285438 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-certs\") pod \"be7d2cc8-ea14-45c9-b096-578619c36bf0\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " Nov 02 15:48:33.285704 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285481 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-data\") pod \"be7d2cc8-ea14-45c9-b096-578619c36bf0\" (UID: \"be7d2cc8-ea14-45c9-b096-578619c36bf0\") " Nov 02 15:48:33.285704 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285507 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-credential-socket" pod "be7d2cc8-ea14-45c9-b096-578619c36bf0" (UID: "be7d2cc8-ea14-45c9-b096-578619c36bf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:33.285704 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285586 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-socket" pod "be7d2cc8-ea14-45c9-b096-578619c36bf0" (UID: "be7d2cc8-ea14-45c9-b096-578619c36bf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:33.285914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285800 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-certs" pod "be7d2cc8-ea14-45c9-b096-578619c36bf0" (UID: "be7d2cc8-ea14-45c9-b096-578619c36bf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:33.285914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285816 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-data" pod "be7d2cc8-ea14-45c9-b096-578619c36bf0" (UID: "be7d2cc8-ea14-45c9-b096-578619c36bf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:33.285914 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285705 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be7d2cc8-ea14-45c9-b096-578619c36bf0-istiod-ca-cert" pod "be7d2cc8-ea14-45c9-b096-578619c36bf0" (UID: "be7d2cc8-ea14-45c9-b096-578619c36bf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:33.286107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285918 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be7d2cc8-ea14-45c9-b096-578619c36bf0-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:33.286107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285940 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:33.286107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285956 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:33.286107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285969 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:33.286107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.285983 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:33.287782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.287759 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-podinfo" pod "be7d2cc8-ea14-45c9-b096-578619c36bf0" (UID: "be7d2cc8-ea14-45c9-b096-578619c36bf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:48:33.287782 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.287774 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-envoy" pod "be7d2cc8-ea14-45c9-b096-578619c36bf0" (UID: "be7d2cc8-ea14-45c9-b096-578619c36bf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:33.288018 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.287973 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-token" pod "be7d2cc8-ea14-45c9-b096-578619c36bf0" (UID: "be7d2cc8-ea14-45c9-b096-578619c36bf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:33.288081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.288035 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-kube-api-access-rfv2z" pod "be7d2cc8-ea14-45c9-b096-578619c36bf0" (UID: "be7d2cc8-ea14-45c9-b096-578619c36bf0"). InnerVolumeSpecName "kube-api-access-rfv2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:33.387561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.387527 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfv2z\" (UniqueName: \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-kube-api-access-rfv2z\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:33.387561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.387559 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:33.387561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.387570 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:33.387818 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.387578 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be7d2cc8-ea14-45c9-b096-578619c36bf0-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:33.442353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.442316 5476 generic.go:356] "Generic (PLEG): container finished" podID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerID="11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42" exitCode=0 Nov 02 15:48:33.442549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.442367 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" Nov 02 15:48:33.442549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.442397 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" event={"ID":"be7d2cc8-ea14-45c9-b096-578619c36bf0","Type":"ContainerDied","Data":"11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42"} Nov 02 15:48:33.442549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.442482 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g" event={"ID":"be7d2cc8-ea14-45c9-b096-578619c36bf0","Type":"ContainerDied","Data":"a63f4c5bcb746a3c06397cb01cc74f2bd5688d37677149f5634b69d913a9abb6"} Nov 02 15:48:33.442549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.442511 5476 scope.go:122] "RemoveContainer" containerID="11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42" Nov 02 15:48:33.453400 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.453377 5476 scope.go:122] "RemoveContainer" containerID="a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462" Nov 02 15:48:33.462522 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.462501 5476 scope.go:122] "RemoveContainer" containerID="11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42" Nov 02 15:48:33.462827 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:48:33.462804 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42\": container with ID starting with 11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42 not found: ID does not exist" containerID="11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42" Nov 02 15:48:33.462882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.462836 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42"} err="failed to get container status \"11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42\": rpc error: code = NotFound desc = could not find container \"11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42\": container with ID starting with 11a04bdb6e004f650d2df43e4f76cdff9a78887fc7ad65165485bfa77ae31a42 not found: ID does not exist" Nov 02 15:48:33.462882 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.462854 5476 scope.go:122] "RemoveContainer" containerID="a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462" Nov 02 15:48:33.463146 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:48:33.463126 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462\": container with ID starting with a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462 not found: ID does not exist" containerID="a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462" Nov 02 15:48:33.463204 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.463153 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462"} err="failed to get container status \"a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462\": rpc error: code = NotFound desc = could not find container \"a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462\": container with ID starting with a3a496fe20578e4834065986ebe8e44979e0c4b64e53feaf1c1f9208ec36c462 not found: ID does not exist" Nov 02 15:48:33.597215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.597173 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g"] Nov 02 15:48:33.601926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.601897 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7ad552-openshift-default-74f9b6dd67-8vn4g"] Nov 02 15:48:33.871964 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.871928 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm"] Nov 02 15:48:33.872225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.872197 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" containerID="cri-o://44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0" gracePeriod=30 Nov 02 15:48:33.924314 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:33.924281 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" path="/var/lib/kubelet/pods/be7d2cc8-ea14-45c9-b096-578619c36bf0/volumes" Nov 02 15:48:39.131075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.131045 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:48:39.251972 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.251883 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-credential-socket\") pod \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " Nov 02 15:48:39.251972 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.251938 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-kube-api-access-n6psl\" (UniqueName: \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-kube-api-access-n6psl\") pod \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " Nov 02 15:48:39.251972 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.251957 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-podinfo\") pod \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " Nov 02 15:48:39.252282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.251979 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-socket\") pod \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " Nov 02 15:48:39.252282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252083 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-certs\") pod \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " Nov 02 15:48:39.252282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252176 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-envoy\") pod \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " Nov 02 15:48:39.252282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252203 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-data\") pod \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " Nov 02 15:48:39.252282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252223 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-credential-socket" pod "b8714e1b-0f48-4e13-afb7-9b9970b88cb3" (UID: "b8714e1b-0f48-4e13-afb7-9b9970b88cb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:39.252282 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252231 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-token\" (UniqueName: \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-token\") pod \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " Nov 02 15:48:39.252595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252319 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istiod-ca-cert\") pod \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\" (UID: \"b8714e1b-0f48-4e13-afb7-9b9970b88cb3\") " Nov 02 15:48:39.252595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252505 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-socket" pod "b8714e1b-0f48-4e13-afb7-9b9970b88cb3" (UID: "b8714e1b-0f48-4e13-afb7-9b9970b88cb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:39.252595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252522 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-certs" pod "b8714e1b-0f48-4e13-afb7-9b9970b88cb3" (UID: "b8714e1b-0f48-4e13-afb7-9b9970b88cb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:39.252595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252534 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-data" pod "b8714e1b-0f48-4e13-afb7-9b9970b88cb3" (UID: "b8714e1b-0f48-4e13-afb7-9b9970b88cb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:39.252872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252823 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istiod-ca-cert" pod "b8714e1b-0f48-4e13-afb7-9b9970b88cb3" (UID: "b8714e1b-0f48-4e13-afb7-9b9970b88cb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:39.252872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252848 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:39.252967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252871 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:39.252967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252887 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:39.252967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.252901 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:39.254300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.254267 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-podinfo" pod "b8714e1b-0f48-4e13-afb7-9b9970b88cb3" (UID: "b8714e1b-0f48-4e13-afb7-9b9970b88cb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:48:39.254409 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.254320 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-envoy" pod "b8714e1b-0f48-4e13-afb7-9b9970b88cb3" (UID: "b8714e1b-0f48-4e13-afb7-9b9970b88cb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:39.254627 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.254605 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-kube-api-access-n6psl" pod "b8714e1b-0f48-4e13-afb7-9b9970b88cb3" (UID: "b8714e1b-0f48-4e13-afb7-9b9970b88cb3"). InnerVolumeSpecName "kube-api-access-n6psl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:39.254751 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.254661 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-token" pod "b8714e1b-0f48-4e13-afb7-9b9970b88cb3" (UID: "b8714e1b-0f48-4e13-afb7-9b9970b88cb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:39.354083 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.354041 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:39.354083 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.354080 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:39.354294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.354094 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:39.354294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.354108 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6psl\" (UniqueName: \"kubernetes.io/projected/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-kube-api-access-n6psl\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:39.354294 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.354123 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8714e1b-0f48-4e13-afb7-9b9970b88cb3-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:39.480368 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.480331 5476 generic.go:356] "Generic (PLEG): container finished" podID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerID="44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0" exitCode=0 Nov 02 15:48:39.480553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.480385 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" Nov 02 15:48:39.480553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.480416 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" event={"ID":"b8714e1b-0f48-4e13-afb7-9b9970b88cb3","Type":"ContainerDied","Data":"44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0"} Nov 02 15:48:39.480553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.480500 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm" event={"ID":"b8714e1b-0f48-4e13-afb7-9b9970b88cb3","Type":"ContainerDied","Data":"b257075ded39b7d121f3fd90c6c7e240fd0925aa775dc7b8aad3ef6019d4f52a"} Nov 02 15:48:39.480553 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.480524 5476 scope.go:122] "RemoveContainer" containerID="44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0" Nov 02 15:48:39.491240 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.491219 5476 scope.go:122] "RemoveContainer" containerID="cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f" Nov 02 15:48:39.500070 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.500049 5476 scope.go:122] "RemoveContainer" containerID="44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0" Nov 02 15:48:39.500349 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:48:39.500328 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0\": container with ID starting with 44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0 not found: ID does not exist" containerID="44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0" Nov 02 15:48:39.500420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.500359 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0"} err="failed to get container status \"44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0\": rpc error: code = NotFound desc = could not find container \"44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0\": container with ID starting with 44a09f66821b01cb5d16a0d2c6eaa8004969051f92b7f89da2a61287f39a82d0 not found: ID does not exist" Nov 02 15:48:39.500420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.500378 5476 scope.go:122] "RemoveContainer" containerID="cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f" Nov 02 15:48:39.500609 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:48:39.500594 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f\": container with ID starting with cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f not found: ID does not exist" containerID="cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f" Nov 02 15:48:39.500656 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.500613 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f"} err="failed to get container status \"cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f\": rpc error: code = NotFound desc = could not find container \"cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f\": container with ID starting with cd27efec1591bccbe931238ce1b2f4c678c3ac5723676a830cee1455eebf961f not found: ID does not exist" Nov 02 15:48:39.635254 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.635222 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm"] Nov 02 15:48:39.640257 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.640228 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-qv2mm"] Nov 02 15:48:39.921033 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:39.920934 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" path="/var/lib/kubelet/pods/b8714e1b-0f48-4e13-afb7-9b9970b88cb3/volumes" Nov 02 15:48:41.542333 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:41.542286 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:48:50.146292 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:48:50.146256 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:48:51.772112 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:51.772072 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4"] Nov 02 15:48:51.772549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:51.772325 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" podUID="ff40bd90-4116-4569-ac57-780ffeee857a" containerName="maas-api" containerID="cri-o://27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d" gracePeriod=30 Nov 02 15:48:52.024359 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.024294 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:48:52.074517 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.074485 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff40bd90-4116-4569-ac57-780ffeee857a-kube-api-access-m98z4\" (UniqueName: \"kubernetes.io/projected/ff40bd90-4116-4569-ac57-780ffeee857a-kube-api-access-m98z4\") pod \"ff40bd90-4116-4569-ac57-780ffeee857a\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " Nov 02 15:48:52.074694 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.074531 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff40bd90-4116-4569-ac57-780ffeee857a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff40bd90-4116-4569-ac57-780ffeee857a-openshift-service-ca\") pod \"ff40bd90-4116-4569-ac57-780ffeee857a\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " Nov 02 15:48:52.074694 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.074593 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-metrics-certs\") pod \"ff40bd90-4116-4569-ac57-780ffeee857a\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " Nov 02 15:48:52.074789 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.074693 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-maas-api-tls\") pod \"ff40bd90-4116-4569-ac57-780ffeee857a\" (UID: \"ff40bd90-4116-4569-ac57-780ffeee857a\") " Nov 02 15:48:52.074939 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.074915 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff40bd90-4116-4569-ac57-780ffeee857a-openshift-service-ca" pod "ff40bd90-4116-4569-ac57-780ffeee857a" (UID: "ff40bd90-4116-4569-ac57-780ffeee857a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:52.075121 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.075101 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff40bd90-4116-4569-ac57-780ffeee857a-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:52.076747 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.076725 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-metrics-certs" pod "ff40bd90-4116-4569-ac57-780ffeee857a" (UID: "ff40bd90-4116-4569-ac57-780ffeee857a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:52.076838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.076762 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-maas-api-tls" pod "ff40bd90-4116-4569-ac57-780ffeee857a" (UID: "ff40bd90-4116-4569-ac57-780ffeee857a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:52.076881 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.076830 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff40bd90-4116-4569-ac57-780ffeee857a-kube-api-access-m98z4" pod "ff40bd90-4116-4569-ac57-780ffeee857a" (UID: "ff40bd90-4116-4569-ac57-780ffeee857a"). InnerVolumeSpecName "kube-api-access-m98z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:52.175838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.175804 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m98z4\" (UniqueName: \"kubernetes.io/projected/ff40bd90-4116-4569-ac57-780ffeee857a-kube-api-access-m98z4\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:52.175838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.175833 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:52.175838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.175842 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff40bd90-4116-4569-ac57-780ffeee857a-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:52.553818 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.553780 5476 generic.go:356] "Generic (PLEG): container finished" podID="ff40bd90-4116-4569-ac57-780ffeee857a" containerID="27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d" exitCode=0 Nov 02 15:48:52.554035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.553834 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" Nov 02 15:48:52.554035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.553864 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" event={"ID":"ff40bd90-4116-4569-ac57-780ffeee857a","Type":"ContainerDied","Data":"27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d"} Nov 02 15:48:52.554035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.553922 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4" event={"ID":"ff40bd90-4116-4569-ac57-780ffeee857a","Type":"ContainerDied","Data":"6511a267ed46978ff8c23525aedaca0ab74c0b0c1fd6a58a7418088ef543cb5c"} Nov 02 15:48:52.554035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.553944 5476 scope.go:122] "RemoveContainer" containerID="27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d" Nov 02 15:48:52.564399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.564375 5476 scope.go:122] "RemoveContainer" containerID="27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d" Nov 02 15:48:52.564730 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:48:52.564708 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d\": container with ID starting with 27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d not found: ID does not exist" containerID="27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d" Nov 02 15:48:52.564808 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.564745 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d"} err="failed to get container status \"27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d\": rpc error: code = NotFound desc = could not find container \"27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d\": container with ID starting with 27c3d6d779b43379659c10fcc0eaf7bf9b435f74bc09b5842557bcd44eda158d not found: ID does not exist" Nov 02 15:48:52.708119 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.708086 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4"] Nov 02 15:48:52.712094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:52.712070 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ff6e1f-67bbfb4685-fzgz4"] Nov 02 15:48:53.921877 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:53.921837 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff40bd90-4116-4569-ac57-780ffeee857a" path="/var/lib/kubelet/pods/ff40bd90-4116-4569-ac57-780ffeee857a/volumes" Nov 02 15:48:57.099566 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:48:57.099533 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:49:02.290789 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.290751 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj"] Nov 02 15:49:02.291220 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291205 5476 cpu_manager.go:414] "containerMap: removing container" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerName="istio-proxy" Nov 02 15:49:02.291220 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291219 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerName="istio-proxy" Nov 02 15:49:02.291319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291229 5476 cpu_manager.go:414] "containerMap: removing container" podUID="d945fed9-7518-4fd5-bb42-ef6d721c1aba" containerName="payload-pre-processing" Nov 02 15:49:02.291319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291234 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d945fed9-7518-4fd5-bb42-ef6d721c1aba" containerName="payload-pre-processing" Nov 02 15:49:02.291319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291253 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" Nov 02 15:49:02.291319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291278 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" Nov 02 15:49:02.291319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291288 5476 cpu_manager.go:414] "containerMap: removing container" podUID="f3570096-5eee-4999-ae44-e3386b5e96f3" containerName="payload-processing" Nov 02 15:49:02.291319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291292 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3570096-5eee-4999-ae44-e3386b5e96f3" containerName="payload-processing" Nov 02 15:49:02.291319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291307 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" Nov 02 15:49:02.291319 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291313 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291324 5476 cpu_manager.go:414] "containerMap: removing container" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291332 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291341 5476 cpu_manager.go:414] "containerMap: removing container" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291348 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291354 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1" containerName="revoke-keys" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291359 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1" containerName="revoke-keys" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291368 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ff40bd90-4116-4569-ac57-780ffeee857a" containerName="maas-api" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291372 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff40bd90-4116-4569-ac57-780ffeee857a" containerName="maas-api" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291379 5476 cpu_manager.go:414] "containerMap: removing container" podUID="eb839896-6a9a-4d11-82a6-597c9d569411" containerName="maas-api" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291382 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb839896-6a9a-4d11-82a6-597c9d569411" containerName="maas-api" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291441 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3570096-5eee-4999-ae44-e3386b5e96f3" containerName="payload-processing" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291451 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291457 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291464 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="be7d2cc8-ea14-45c9-b096-578619c36bf0" containerName="istio-proxy" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291469 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1" containerName="revoke-keys" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291474 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff40bd90-4116-4569-ac57-780ffeee857a" containerName="maas-api" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291483 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb839896-6a9a-4d11-82a6-597c9d569411" containerName="maas-api" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291495 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8cd888-ff5b-44a2-b45d-df67e4cf288c" containerName="istio-proxy" Nov 02 15:49:02.291614 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.291502 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d945fed9-7518-4fd5-bb42-ef6d721c1aba" containerName="payload-pre-processing" Nov 02 15:49:02.296379 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.296357 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:02.375211 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.375168 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggh4w\" (UniqueName: \"kubernetes.io/projected/e976a2e0-982c-45d0-abc9-31cf8cdade22-kube-api-access-ggh4w\") pod \"maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:02.375423 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.375226 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:02.375423 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.375324 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:02.393654 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.393621 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:49:02.393654 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.393642 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:49:02.393931 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.393622 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:49:02.393931 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.393680 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:49:02.395957 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.395934 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj"] Nov 02 15:49:02.476517 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.476475 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggh4w\" (UniqueName: \"kubernetes.io/projected/e976a2e0-982c-45d0-abc9-31cf8cdade22-kube-api-access-ggh4w\") pod \"maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:02.476737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.476538 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:02.476737 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.476583 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:02.477514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.477487 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:02.480384 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.480356 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:02.535039 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.534984 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggh4w\" (UniqueName: \"kubernetes.io/projected/e976a2e0-982c-45d0-abc9-31cf8cdade22-kube-api-access-ggh4w\") pod \"maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:02.609644 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.609611 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:02.985479 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:02.985398 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj"] Nov 02 15:49:02.991694 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:49:02.991664 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode976a2e0_982c_45d0_abc9_31cf8cdade22.slice/crio-d2a8cdb1e41527f71187d7f517de669ef6ded2a7944b43a67311626ccf315179 WatchSource:0}: Error finding container d2a8cdb1e41527f71187d7f517de669ef6ded2a7944b43a67311626ccf315179: Status 404 returned error can't find the container with id d2a8cdb1e41527f71187d7f517de669ef6ded2a7944b43a67311626ccf315179 Nov 02 15:49:03.619789 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:03.619740 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" event={"ID":"e976a2e0-982c-45d0-abc9-31cf8cdade22","Type":"ContainerStarted","Data":"b05e8ec22461f77b5ff4e11712cb0121478e95b0b875f9e3bbbcb50f9f76c031"} Nov 02 15:49:03.620247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:03.619830 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" event={"ID":"e976a2e0-982c-45d0-abc9-31cf8cdade22","Type":"ContainerStarted","Data":"d2a8cdb1e41527f71187d7f517de669ef6ded2a7944b43a67311626ccf315179"} Nov 02 15:49:03.729936 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:03.729875 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" podStartSLOduration=1.729860356 podStartE2EDuration="1.729860356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:03.72855039 +0000 UTC m=+3045.931428448" watchObservedRunningTime="2026-11-02 15:49:03.729860356 +0000 UTC m=+3045.932738414" Nov 02 15:49:04.626895 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:04.626854 5476 generic.go:356] "Generic (PLEG): container finished" podID="e976a2e0-982c-45d0-abc9-31cf8cdade22" containerID="b05e8ec22461f77b5ff4e11712cb0121478e95b0b875f9e3bbbcb50f9f76c031" exitCode=0 Nov 02 15:49:04.627362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:04.626932 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" event={"ID":"e976a2e0-982c-45d0-abc9-31cf8cdade22","Type":"ContainerDied","Data":"b05e8ec22461f77b5ff4e11712cb0121478e95b0b875f9e3bbbcb50f9f76c031"} Nov 02 15:49:05.767523 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:05.767495 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:05.909365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:05.909275 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-xks-ca\") pod \"e976a2e0-982c-45d0-abc9-31cf8cdade22\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " Nov 02 15:49:05.909365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:05.909344 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-openshift-ca\") pod \"e976a2e0-982c-45d0-abc9-31cf8cdade22\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " Nov 02 15:49:05.909612 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:05.909382 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e976a2e0-982c-45d0-abc9-31cf8cdade22-kube-api-access-ggh4w\" (UniqueName: \"kubernetes.io/projected/e976a2e0-982c-45d0-abc9-31cf8cdade22-kube-api-access-ggh4w\") pod \"e976a2e0-982c-45d0-abc9-31cf8cdade22\" (UID: \"e976a2e0-982c-45d0-abc9-31cf8cdade22\") " Nov 02 15:49:05.909791 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:05.909766 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-openshift-ca" pod "e976a2e0-982c-45d0-abc9-31cf8cdade22" (UID: "e976a2e0-982c-45d0-abc9-31cf8cdade22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:49:05.911496 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:05.911471 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e976a2e0-982c-45d0-abc9-31cf8cdade22-kube-api-access-ggh4w" pod "e976a2e0-982c-45d0-abc9-31cf8cdade22" (UID: "e976a2e0-982c-45d0-abc9-31cf8cdade22"). InnerVolumeSpecName "kube-api-access-ggh4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:05.911606 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:05.911498 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-xks-ca" pod "e976a2e0-982c-45d0-abc9-31cf8cdade22" (UID: "e976a2e0-982c-45d0-abc9-31cf8cdade22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:06.010532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:06.010481 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.010532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:06.010524 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e976a2e0-982c-45d0-abc9-31cf8cdade22-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.010532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:06.010536 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggh4w\" (UniqueName: \"kubernetes.io/projected/e976a2e0-982c-45d0-abc9-31cf8cdade22-kube-api-access-ggh4w\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:06.640870 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:06.640840 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" Nov 02 15:49:06.640870 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:06.640853 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj" event={"ID":"e976a2e0-982c-45d0-abc9-31cf8cdade22","Type":"ContainerDied","Data":"d2a8cdb1e41527f71187d7f517de669ef6ded2a7944b43a67311626ccf315179"} Nov 02 15:49:06.641094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:06.640912 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2a8cdb1e41527f71187d7f517de669ef6ded2a7944b43a67311626ccf315179" Nov 02 15:49:06.666947 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:06.666909 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f"] Nov 02 15:49:13.738934 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:13.738898 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d"] Nov 02 15:49:14.087388 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.087306 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj"] Nov 02 15:49:14.087797 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.087782 5476 cpu_manager.go:414] "containerMap: removing container" podUID="e976a2e0-982c-45d0-abc9-31cf8cdade22" containerName="revoke-keys" Nov 02 15:49:14.087841 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.087799 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e976a2e0-982c-45d0-abc9-31cf8cdade22" containerName="revoke-keys" Nov 02 15:49:14.087892 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.087883 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e976a2e0-982c-45d0-abc9-31cf8cdade22" containerName="revoke-keys" Nov 02 15:49:14.087928 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.087895 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8714e1b-0f48-4e13-afb7-9b9970b88cb3" containerName="istio-proxy" Nov 02 15:49:14.090824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.090803 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:14.140470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.140407 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:49:14.141232 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.141205 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:49:14.141413 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.141301 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:49:14.141413 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.141384 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:49:14.153338 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.153299 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj"] Nov 02 15:49:14.192430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.192391 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa11236-g22tj\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:14.192625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.192463 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa11236-g22tj\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:14.192625 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.192595 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzgp\" (UniqueName: \"kubernetes.io/projected/f2ea013a-bfb2-46c4-a837-99d864255101-kube-api-access-pvzgp\") pod \"maas-api-revoke-sub-aaa11236-g22tj\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:14.293537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.293495 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa11236-g22tj\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:14.293728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.293582 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa11236-g22tj\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:14.293728 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.293630 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzgp\" (UniqueName: \"kubernetes.io/projected/f2ea013a-bfb2-46c4-a837-99d864255101-kube-api-access-pvzgp\") pod \"maas-api-revoke-sub-aaa11236-g22tj\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:14.294247 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.294218 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa11236-g22tj\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:14.295884 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.295865 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa11236-g22tj\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:14.351134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.351066 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzgp\" (UniqueName: \"kubernetes.io/projected/f2ea013a-bfb2-46c4-a837-99d864255101-kube-api-access-pvzgp\") pod \"maas-api-revoke-sub-aaa11236-g22tj\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:14.403441 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.403404 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:14.773504 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:14.773470 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj"] Nov 02 15:49:14.779452 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:49:14.779420 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ea013a_bfb2_46c4_a837_99d864255101.slice/crio-68093899e56fdbbe1d2b708ac7df6cc7289a5cbb3380c9eb69f9944e95e48c47 WatchSource:0}: Error finding container 68093899e56fdbbe1d2b708ac7df6cc7289a5cbb3380c9eb69f9944e95e48c47: Status 404 returned error can't find the container with id 68093899e56fdbbe1d2b708ac7df6cc7289a5cbb3380c9eb69f9944e95e48c47 Nov 02 15:49:15.694662 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:15.694624 5476 generic.go:356] "Generic (PLEG): container finished" podID="f2ea013a-bfb2-46c4-a837-99d864255101" containerID="3fd238b318fc07f562632037657d1db5225577b4182c0e5166f7333ab04e3dee" exitCode=0 Nov 02 15:49:15.694838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:15.694659 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" event={"ID":"f2ea013a-bfb2-46c4-a837-99d864255101","Type":"ContainerDied","Data":"3fd238b318fc07f562632037657d1db5225577b4182c0e5166f7333ab04e3dee"} Nov 02 15:49:15.694838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:15.694715 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" event={"ID":"f2ea013a-bfb2-46c4-a837-99d864255101","Type":"ContainerStarted","Data":"68093899e56fdbbe1d2b708ac7df6cc7289a5cbb3380c9eb69f9944e95e48c47"} Nov 02 15:49:16.835608 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:16.835580 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:17.021982 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.021897 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ea013a-bfb2-46c4-a837-99d864255101-kube-api-access-pvzgp\" (UniqueName: \"kubernetes.io/projected/f2ea013a-bfb2-46c4-a837-99d864255101-kube-api-access-pvzgp\") pod \"f2ea013a-bfb2-46c4-a837-99d864255101\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " Nov 02 15:49:17.021982 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.021940 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-openshift-ca\") pod \"f2ea013a-bfb2-46c4-a837-99d864255101\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " Nov 02 15:49:17.021982 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.021978 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-xks-ca\") pod \"f2ea013a-bfb2-46c4-a837-99d864255101\" (UID: \"f2ea013a-bfb2-46c4-a837-99d864255101\") " Nov 02 15:49:17.022399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.022374 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-openshift-ca" pod "f2ea013a-bfb2-46c4-a837-99d864255101" (UID: "f2ea013a-bfb2-46c4-a837-99d864255101"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:49:17.024197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.024171 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ea013a-bfb2-46c4-a837-99d864255101-kube-api-access-pvzgp" pod "f2ea013a-bfb2-46c4-a837-99d864255101" (UID: "f2ea013a-bfb2-46c4-a837-99d864255101"). InnerVolumeSpecName "kube-api-access-pvzgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:17.024197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.024171 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-xks-ca" pod "f2ea013a-bfb2-46c4-a837-99d864255101" (UID: "f2ea013a-bfb2-46c4-a837-99d864255101"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:17.123615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.123574 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:17.123615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.123608 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvzgp\" (UniqueName: \"kubernetes.io/projected/f2ea013a-bfb2-46c4-a837-99d864255101-kube-api-access-pvzgp\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:17.123615 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.123621 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2ea013a-bfb2-46c4-a837-99d864255101-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:17.708149 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.708111 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" Nov 02 15:49:17.708335 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.708109 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj" event={"ID":"f2ea013a-bfb2-46c4-a837-99d864255101","Type":"ContainerDied","Data":"68093899e56fdbbe1d2b708ac7df6cc7289a5cbb3380c9eb69f9944e95e48c47"} Nov 02 15:49:17.708335 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:17.708241 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68093899e56fdbbe1d2b708ac7df6cc7289a5cbb3380c9eb69f9944e95e48c47" Nov 02 15:49:18.740066 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:18.739988 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" podUID="4829d867-65f5-4784-a4fb-95d488ac9bcf" containerName="payload-processing" containerID="cri-o://9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d" gracePeriod=25 Nov 02 15:49:19.002835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.002769 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:49:19.145836 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.145797 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4829d867-65f5-4784-a4fb-95d488ac9bcf-kube-api-access-4gdbq\" (UniqueName: \"kubernetes.io/projected/4829d867-65f5-4784-a4fb-95d488ac9bcf-kube-api-access-4gdbq\") pod \"4829d867-65f5-4784-a4fb-95d488ac9bcf\" (UID: \"4829d867-65f5-4784-a4fb-95d488ac9bcf\") " Nov 02 15:49:19.146030 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.145959 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4829d867-65f5-4784-a4fb-95d488ac9bcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4829d867-65f5-4784-a4fb-95d488ac9bcf-plugins-config-volume\") pod \"4829d867-65f5-4784-a4fb-95d488ac9bcf\" (UID: \"4829d867-65f5-4784-a4fb-95d488ac9bcf\") " Nov 02 15:49:19.146318 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.146296 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4829d867-65f5-4784-a4fb-95d488ac9bcf-plugins-config-volume" pod "4829d867-65f5-4784-a4fb-95d488ac9bcf" (UID: "4829d867-65f5-4784-a4fb-95d488ac9bcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:49:19.148124 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.148091 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4829d867-65f5-4784-a4fb-95d488ac9bcf-kube-api-access-4gdbq" pod "4829d867-65f5-4784-a4fb-95d488ac9bcf" (UID: "4829d867-65f5-4784-a4fb-95d488ac9bcf"). InnerVolumeSpecName "kube-api-access-4gdbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:19.247576 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.247533 5476 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4829d867-65f5-4784-a4fb-95d488ac9bcf-plugins-config-volume\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:19.247576 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.247564 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gdbq\" (UniqueName: \"kubernetes.io/projected/4829d867-65f5-4784-a4fb-95d488ac9bcf-kube-api-access-4gdbq\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:19.721654 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.721618 5476 generic.go:356] "Generic (PLEG): container finished" podID="4829d867-65f5-4784-a4fb-95d488ac9bcf" containerID="9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d" exitCode=0 Nov 02 15:49:19.721838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.721661 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" Nov 02 15:49:19.721838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.721698 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" event={"ID":"4829d867-65f5-4784-a4fb-95d488ac9bcf","Type":"ContainerDied","Data":"9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d"} Nov 02 15:49:19.721838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.721758 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d" event={"ID":"4829d867-65f5-4784-a4fb-95d488ac9bcf","Type":"ContainerDied","Data":"d06f86531c9e3cf836827dac6380d00adecedc68256e97102b40cbc6d9497811"} Nov 02 15:49:19.721838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.721782 5476 scope.go:122] "RemoveContainer" containerID="9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d" Nov 02 15:49:19.733215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.733190 5476 scope.go:122] "RemoveContainer" containerID="9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d" Nov 02 15:49:19.733532 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:49:19.733510 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d\": container with ID starting with 9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d not found: ID does not exist" containerID="9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d" Nov 02 15:49:19.733638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.733542 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d"} err="failed to get container status \"9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d\": rpc error: code = NotFound desc = could not find container \"9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d\": container with ID starting with 9bbd6815459975507daae58cd14511d93b85e2684f8f6d1b7383abc5638c073d not found: ID does not exist" Nov 02 15:49:19.881541 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.881506 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d"] Nov 02 15:49:19.927769 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:19.927738 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fce776-64cc9469dc-4m29d"] Nov 02 15:49:20.146803 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:49:20.146758 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:49:21.688480 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:21.688412 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" containerID="cri-o://709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba" gracePeriod=45 Nov 02 15:49:21.836465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:21.836423 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": dial tcp 10.134.0.91:8000: connect: connection refused" Nov 02 15:49:21.922248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:21.922219 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4829d867-65f5-4784-a4fb-95d488ac9bcf" path="/var/lib/kubelet/pods/4829d867-65f5-4784-a4fb-95d488ac9bcf/volumes" Nov 02 15:49:21.939183 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:21.939100 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:49:22.073597 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.073561 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-model-cache\") pod \"82575b53-acff-42c1-be4c-b863202c8f92\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " Nov 02 15:49:22.073790 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.073670 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-dshm\") pod \"82575b53-acff-42c1-be4c-b863202c8f92\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " Nov 02 15:49:22.073790 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.073718 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-home\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-home\") pod \"82575b53-acff-42c1-be4c-b863202c8f92\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " Nov 02 15:49:22.073790 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.073758 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82575b53-acff-42c1-be4c-b863202c8f92-tls-certs\" (UniqueName: \"kubernetes.io/secret/82575b53-acff-42c1-be4c-b863202c8f92-tls-certs\") pod \"82575b53-acff-42c1-be4c-b863202c8f92\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " Nov 02 15:49:22.073937 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.073811 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-tmp-dir\") pod \"82575b53-acff-42c1-be4c-b863202c8f92\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " Nov 02 15:49:22.073937 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.073844 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82575b53-acff-42c1-be4c-b863202c8f92-kube-api-access-9dgz6\" (UniqueName: \"kubernetes.io/projected/82575b53-acff-42c1-be4c-b863202c8f92-kube-api-access-9dgz6\") pod \"82575b53-acff-42c1-be4c-b863202c8f92\" (UID: \"82575b53-acff-42c1-be4c-b863202c8f92\") " Nov 02 15:49:22.073937 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.073885 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-model-cache" pod "82575b53-acff-42c1-be4c-b863202c8f92" (UID: "82575b53-acff-42c1-be4c-b863202c8f92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.074411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.074006 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-home" pod "82575b53-acff-42c1-be4c-b863202c8f92" (UID: "82575b53-acff-42c1-be4c-b863202c8f92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.074411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.074224 5476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-model-cache\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.074411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.074274 5476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-home\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.074411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.074311 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-tmp-dir" pod "82575b53-acff-42c1-be4c-b863202c8f92" (UID: "82575b53-acff-42c1-be4c-b863202c8f92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.079034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.076373 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82575b53-acff-42c1-be4c-b863202c8f92-kube-api-access-9dgz6" pod "82575b53-acff-42c1-be4c-b863202c8f92" (UID: "82575b53-acff-42c1-be4c-b863202c8f92"). InnerVolumeSpecName "kube-api-access-9dgz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:22.079034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.076648 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82575b53-acff-42c1-be4c-b863202c8f92-tls-certs" pod "82575b53-acff-42c1-be4c-b863202c8f92" (UID: "82575b53-acff-42c1-be4c-b863202c8f92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:22.079034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.076695 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-dshm" pod "82575b53-acff-42c1-be4c-b863202c8f92" (UID: "82575b53-acff-42c1-be4c-b863202c8f92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.175286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.175249 5476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-dshm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.175286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.175283 5476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82575b53-acff-42c1-be4c-b863202c8f92-tls-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.175509 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.175311 5476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82575b53-acff-42c1-be4c-b863202c8f92-tmp-dir\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.175509 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.175321 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dgz6\" (UniqueName: \"kubernetes.io/projected/82575b53-acff-42c1-be4c-b863202c8f92-kube-api-access-9dgz6\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.742025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.741959 5476 generic.go:356] "Generic (PLEG): container finished" podID="82575b53-acff-42c1-be4c-b863202c8f92" containerID="709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba" exitCode=0 Nov 02 15:49:22.742025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.742019 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" Nov 02 15:49:22.742590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.742050 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" event={"ID":"82575b53-acff-42c1-be4c-b863202c8f92","Type":"ContainerDied","Data":"709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba"} Nov 02 15:49:22.742590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.742101 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f" event={"ID":"82575b53-acff-42c1-be4c-b863202c8f92","Type":"ContainerDied","Data":"a5951f9a4bd3786dc916ef6f1d9a74091caca0fceba291bed76988a45368c599"} Nov 02 15:49:22.742590 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.742124 5476 scope.go:122] "RemoveContainer" containerID="709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba" Nov 02 15:49:22.752136 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.752113 5476 scope.go:122] "RemoveContainer" containerID="709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba" Nov 02 15:49:22.752415 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:49:22.752394 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba\": container with ID starting with 709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba not found: ID does not exist" containerID="709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba" Nov 02 15:49:22.752475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.752423 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba"} err="failed to get container status \"709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba\": rpc error: code = NotFound desc = could not find container \"709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba\": container with ID starting with 709934fe13e9cb03721f5e10ee9a04c79f1c7dac29f5a2ae7fb1b9fd23fd12ba not found: ID does not exist" Nov 02 15:49:22.895872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.895836 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f"] Nov 02 15:49:22.942072 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:22.942031 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-2-w2-fce776-kserve-776d95b978-vgz5f"] Nov 02 15:49:23.921409 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:23.921373 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82575b53-acff-42c1-be4c-b863202c8f92" path="/var/lib/kubelet/pods/82575b53-acff-42c1-be4c-b863202c8f92/volumes" Nov 02 15:49:30.560195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:30.560110 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:49:31.543528 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:31.543490 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj"] Nov 02 15:49:31.589635 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:31.589597 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fce776-9mvbj"] Nov 02 15:49:31.920634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:31.920597 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e976a2e0-982c-45d0-abc9-31cf8cdade22" path="/var/lib/kubelet/pods/e976a2e0-982c-45d0-abc9-31cf8cdade22/volumes" Nov 02 15:49:45.356552 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:49:45.356514 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:49:50.146701 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:49:50.146660 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:50:07.532834 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.532798 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4"] Nov 02 15:50:07.533353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.533190 5476 cpu_manager.go:414] "containerMap: removing container" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" Nov 02 15:50:07.533353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.533200 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" Nov 02 15:50:07.533353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.533230 5476 cpu_manager.go:414] "containerMap: removing container" podUID="4829d867-65f5-4784-a4fb-95d488ac9bcf" containerName="payload-processing" Nov 02 15:50:07.533353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.533237 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4829d867-65f5-4784-a4fb-95d488ac9bcf" containerName="payload-processing" Nov 02 15:50:07.533353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.533255 5476 cpu_manager.go:414] "containerMap: removing container" podUID="f2ea013a-bfb2-46c4-a837-99d864255101" containerName="revoke-keys" Nov 02 15:50:07.533353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.533261 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ea013a-bfb2-46c4-a837-99d864255101" containerName="revoke-keys" Nov 02 15:50:07.533353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.533320 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4829d867-65f5-4784-a4fb-95d488ac9bcf" containerName="payload-processing" Nov 02 15:50:07.533353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.533333 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ea013a-bfb2-46c4-a837-99d864255101" containerName="revoke-keys" Nov 02 15:50:07.533353 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.533341 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="82575b53-acff-42c1-be4c-b863202c8f92" containerName="main" Nov 02 15:50:07.535967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.535949 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:07.584942 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.584910 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:50:07.617791 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.617738 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljnft\" (UniqueName: \"kubernetes.io/projected/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-kube-api-access-ljnft\") pod \"maas-api-revoke-sub-85237e45-bvbl4\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:07.618015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.617795 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bvbl4\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:07.618113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.618019 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bvbl4\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:07.631026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.630979 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:50:07.631026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.630979 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:50:07.631245 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.631033 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:50:07.633261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.633240 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4"] Nov 02 15:50:07.718878 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.718845 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bvbl4\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:07.719106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.718917 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljnft\" (UniqueName: \"kubernetes.io/projected/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-kube-api-access-ljnft\") pod \"maas-api-revoke-sub-85237e45-bvbl4\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:07.719106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.718938 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bvbl4\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:07.719557 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.719534 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-bvbl4\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:07.721365 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.721342 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-bvbl4\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:07.776804 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.776773 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljnft\" (UniqueName: \"kubernetes.io/projected/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-kube-api-access-ljnft\") pod \"maas-api-revoke-sub-85237e45-bvbl4\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:07.849212 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:07.849124 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:08.219810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:08.219767 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4"] Nov 02 15:50:08.225823 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:50:08.225789 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f41fe48_373c_4a7d_bf00_aed6496ac6a2.slice/crio-6dd74835435c85f496235c2fddd76491a6295b66b7965bb23acbf8980e1ce56e WatchSource:0}: Error finding container 6dd74835435c85f496235c2fddd76491a6295b66b7965bb23acbf8980e1ce56e: Status 404 returned error can't find the container with id 6dd74835435c85f496235c2fddd76491a6295b66b7965bb23acbf8980e1ce56e Nov 02 15:50:09.000646 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:09.000608 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" event={"ID":"7f41fe48-373c-4a7d-bf00-aed6496ac6a2","Type":"ContainerStarted","Data":"d61f9917a614bada63af490f9f16f7988a29700baad7cf9dc3497d0a7204c688"} Nov 02 15:50:09.001086 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:09.000669 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" event={"ID":"7f41fe48-373c-4a7d-bf00-aed6496ac6a2","Type":"ContainerStarted","Data":"6dd74835435c85f496235c2fddd76491a6295b66b7965bb23acbf8980e1ce56e"} Nov 02 15:50:09.110263 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:09.110200 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" podStartSLOduration=2.110186204 podStartE2EDuration="2.110186204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:09.109293039 +0000 UTC m=+3111.312171099" watchObservedRunningTime="2026-11-02 15:50:09.110186204 +0000 UTC m=+3111.313064264" Nov 02 15:50:10.008142 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:10.008104 5476 generic.go:356] "Generic (PLEG): container finished" podID="7f41fe48-373c-4a7d-bf00-aed6496ac6a2" containerID="d61f9917a614bada63af490f9f16f7988a29700baad7cf9dc3497d0a7204c688" exitCode=0 Nov 02 15:50:10.008558 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:10.008189 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" event={"ID":"7f41fe48-373c-4a7d-bf00-aed6496ac6a2","Type":"ContainerDied","Data":"d61f9917a614bada63af490f9f16f7988a29700baad7cf9dc3497d0a7204c688"} Nov 02 15:50:11.155029 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:11.154988 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:11.253153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:11.253114 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-openshift-ca\") pod \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " Nov 02 15:50:11.253326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:11.253186 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-xks-ca\") pod \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " Nov 02 15:50:11.253326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:11.253207 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-kube-api-access-ljnft\" (UniqueName: \"kubernetes.io/projected/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-kube-api-access-ljnft\") pod \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\" (UID: \"7f41fe48-373c-4a7d-bf00-aed6496ac6a2\") " Nov 02 15:50:11.253526 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:11.253502 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-openshift-ca" pod "7f41fe48-373c-4a7d-bf00-aed6496ac6a2" (UID: "7f41fe48-373c-4a7d-bf00-aed6496ac6a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:50:11.255346 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:11.255320 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-xks-ca" pod "7f41fe48-373c-4a7d-bf00-aed6496ac6a2" (UID: "7f41fe48-373c-4a7d-bf00-aed6496ac6a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:50:11.255467 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:11.255355 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-kube-api-access-ljnft" pod "7f41fe48-373c-4a7d-bf00-aed6496ac6a2" (UID: "7f41fe48-373c-4a7d-bf00-aed6496ac6a2"). InnerVolumeSpecName "kube-api-access-ljnft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:50:11.354198 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:11.354165 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:50:11.354366 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:11.354200 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljnft\" (UniqueName: \"kubernetes.io/projected/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-kube-api-access-ljnft\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:50:11.354366 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:11.354226 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f41fe48-373c-4a7d-bf00-aed6496ac6a2-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:50:12.022755 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:12.022716 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" event={"ID":"7f41fe48-373c-4a7d-bf00-aed6496ac6a2","Type":"ContainerDied","Data":"6dd74835435c85f496235c2fddd76491a6295b66b7965bb23acbf8980e1ce56e"} Nov 02 15:50:12.022934 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:12.022781 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dd74835435c85f496235c2fddd76491a6295b66b7965bb23acbf8980e1ce56e" Nov 02 15:50:12.022934 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:12.022736 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4" Nov 02 15:50:20.147233 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:50:20.147182 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:50:37.046680 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:37.046642 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph"] Nov 02 15:50:37.092805 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:37.092766 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03db5bb4-rhsph"] Nov 02 15:50:37.920563 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:50:37.920521 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8777e2c-5138-42a2-88ea-2426d7de5ea1" path="/var/lib/kubelet/pods/b8777e2c-5138-42a2-88ea-2426d7de5ea1/volumes" Nov 02 15:50:50.146694 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:50:50.146654 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:51:06.975759 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:06.975724 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:51:15.537416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.537386 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw"] Nov 02 15:51:15.537826 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.537781 5476 cpu_manager.go:414] "containerMap: removing container" podUID="7f41fe48-373c-4a7d-bf00-aed6496ac6a2" containerName="revoke-keys" Nov 02 15:51:15.537826 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.537792 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f41fe48-373c-4a7d-bf00-aed6496ac6a2" containerName="revoke-keys" Nov 02 15:51:15.537920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.537864 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f41fe48-373c-4a7d-bf00-aed6496ac6a2" containerName="revoke-keys" Nov 02 15:51:15.541464 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.541445 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:15.579681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.579647 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-f4fnw\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:15.579830 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.579701 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6wkr\" (UniqueName: \"kubernetes.io/projected/afa6a36d-087b-4135-8e9a-361365158a21-kube-api-access-f6wkr\") pod \"maas-api-revoke-sub-c1e271a9-f4fnw\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:15.579830 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.579791 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-f4fnw\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:15.590382 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.590355 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:51:15.590517 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.590424 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:51:15.590517 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.590438 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:51:15.636592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.636559 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:51:15.638634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.638612 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw"] Nov 02 15:51:15.681032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.680982 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-f4fnw\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:15.681196 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.681065 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6wkr\" (UniqueName: \"kubernetes.io/projected/afa6a36d-087b-4135-8e9a-361365158a21-kube-api-access-f6wkr\") pod \"maas-api-revoke-sub-c1e271a9-f4fnw\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:15.681196 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.681118 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-f4fnw\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:15.681746 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.681723 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-f4fnw\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:15.683588 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.683560 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-f4fnw\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:15.739571 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.739542 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6wkr\" (UniqueName: \"kubernetes.io/projected/afa6a36d-087b-4135-8e9a-361365158a21-kube-api-access-f6wkr\") pod \"maas-api-revoke-sub-c1e271a9-f4fnw\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:15.854690 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:15.854663 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:16.226122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:16.226035 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw"] Nov 02 15:51:16.232068 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:51:16.232036 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa6a36d_087b_4135_8e9a_361365158a21.slice/crio-5a862ccf463dcf7db83088ab3787e4c8b9380fa6db563c2adb8ab29a24d2fcdc WatchSource:0}: Error finding container 5a862ccf463dcf7db83088ab3787e4c8b9380fa6db563c2adb8ab29a24d2fcdc: Status 404 returned error can't find the container with id 5a862ccf463dcf7db83088ab3787e4c8b9380fa6db563c2adb8ab29a24d2fcdc Nov 02 15:51:16.383961 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:16.383918 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" event={"ID":"afa6a36d-087b-4135-8e9a-361365158a21","Type":"ContainerStarted","Data":"5a862ccf463dcf7db83088ab3787e4c8b9380fa6db563c2adb8ab29a24d2fcdc"} Nov 02 15:51:17.390447 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:17.390410 5476 generic.go:356] "Generic (PLEG): container finished" podID="afa6a36d-087b-4135-8e9a-361365158a21" containerID="97e1f761708c0d889a5eb8ce73f377b2f6d2e9cf01ba5a4601ea2aa32b2101dd" exitCode=0 Nov 02 15:51:17.390923 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:17.390500 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" event={"ID":"afa6a36d-087b-4135-8e9a-361365158a21","Type":"ContainerDied","Data":"97e1f761708c0d889a5eb8ce73f377b2f6d2e9cf01ba5a4601ea2aa32b2101dd"} Nov 02 15:51:18.539800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:18.539774 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:18.605182 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:18.605156 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-xks-ca\") pod \"afa6a36d-087b-4135-8e9a-361365158a21\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " Nov 02 15:51:18.605329 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:18.605200 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-openshift-ca\") pod \"afa6a36d-087b-4135-8e9a-361365158a21\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " Nov 02 15:51:18.605329 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:18.605230 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa6a36d-087b-4135-8e9a-361365158a21-kube-api-access-f6wkr\" (UniqueName: \"kubernetes.io/projected/afa6a36d-087b-4135-8e9a-361365158a21-kube-api-access-f6wkr\") pod \"afa6a36d-087b-4135-8e9a-361365158a21\" (UID: \"afa6a36d-087b-4135-8e9a-361365158a21\") " Nov 02 15:51:18.605509 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:18.605485 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-openshift-ca" pod "afa6a36d-087b-4135-8e9a-361365158a21" (UID: "afa6a36d-087b-4135-8e9a-361365158a21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:51:18.607234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:18.607202 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-xks-ca" pod "afa6a36d-087b-4135-8e9a-361365158a21" (UID: "afa6a36d-087b-4135-8e9a-361365158a21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:51:18.607417 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:18.607394 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa6a36d-087b-4135-8e9a-361365158a21-kube-api-access-f6wkr" pod "afa6a36d-087b-4135-8e9a-361365158a21" (UID: "afa6a36d-087b-4135-8e9a-361365158a21"). InnerVolumeSpecName "kube-api-access-f6wkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:51:18.706519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:18.706453 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:51:18.706519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:18.706482 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa6a36d-087b-4135-8e9a-361365158a21-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:51:18.706519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:18.706492 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6wkr\" (UniqueName: \"kubernetes.io/projected/afa6a36d-087b-4135-8e9a-361365158a21-kube-api-access-f6wkr\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:51:19.402172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:19.402145 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" Nov 02 15:51:19.402347 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:19.402145 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw" event={"ID":"afa6a36d-087b-4135-8e9a-361365158a21","Type":"ContainerDied","Data":"5a862ccf463dcf7db83088ab3787e4c8b9380fa6db563c2adb8ab29a24d2fcdc"} Nov 02 15:51:19.402347 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:19.402264 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a862ccf463dcf7db83088ab3787e4c8b9380fa6db563c2adb8ab29a24d2fcdc" Nov 02 15:51:19.812810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:19.812731 5476 scope.go:122] "RemoveContainer" containerID="4e28f2f6585f33331c7c03ea0286176c3e4a3daeadd7d39d5ba67e4641ae5d4b" Nov 02 15:51:20.145266 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:51:20.145232 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:51:28.387264 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:28.387231 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:51:37.805107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.805072 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp"] Nov 02 15:51:37.805742 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.805468 5476 cpu_manager.go:414] "containerMap: removing container" podUID="afa6a36d-087b-4135-8e9a-361365158a21" containerName="revoke-keys" Nov 02 15:51:37.805742 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.805479 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa6a36d-087b-4135-8e9a-361365158a21" containerName="revoke-keys" Nov 02 15:51:37.805742 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.805559 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa6a36d-087b-4135-8e9a-361365158a21" containerName="revoke-keys" Nov 02 15:51:37.809962 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.809942 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.885256 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.885229 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.885396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.885282 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee63d99-1900-45ed-a48d-64d2734fef42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.885396 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.885300 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.885498 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.885423 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.885498 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.885447 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.885498 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.885489 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.885643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.885594 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.885643 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.885620 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52qnf\" (UniqueName: \"kubernetes.io/projected/9ee63d99-1900-45ed-a48d-64d2734fef42-kube-api-access-52qnf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.885712 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.885644 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.986979 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987043 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987265 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987079 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987265 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987126 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987265 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987150 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52qnf\" (UniqueName: \"kubernetes.io/projected/9ee63d99-1900-45ed-a48d-64d2734fef42-kube-api-access-52qnf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987265 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987176 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987265 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987233 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987530 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987313 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee63d99-1900-45ed-a48d-64d2734fef42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987530 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987341 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987530 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987439 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987530 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987482 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987534 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.987686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.987663 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.988085 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.988067 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ee63d99-1900-45ed-a48d-64d2734fef42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.989401 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.989379 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:37.989663 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:37.989644 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:38.019767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:38.019741 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp"] Nov 02 15:51:38.054977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:38.054952 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ee63d99-1900-45ed-a48d-64d2734fef42-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:38.055133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:38.055031 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52qnf\" (UniqueName: \"kubernetes.io/projected/9ee63d99-1900-45ed-a48d-64d2734fef42-kube-api-access-52qnf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-wmnbp\" (UID: \"9ee63d99-1900-45ed-a48d-64d2734fef42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:38.122935 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:38.122911 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:38.485666 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:38.485634 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:51:38.548565 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:38.548526 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp"] Nov 02 15:51:38.553950 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:51:38.553915 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee63d99_1900_45ed_a48d_64d2734fef42.slice/crio-a755c15f9d075cdc795d06d6f22afbb1ae62e166aedb1be716a02d4844fe8017 WatchSource:0}: Error finding container a755c15f9d075cdc795d06d6f22afbb1ae62e166aedb1be716a02d4844fe8017: Status 404 returned error can't find the container with id a755c15f9d075cdc795d06d6f22afbb1ae62e166aedb1be716a02d4844fe8017 Nov 02 15:51:38.556040 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:38.556013 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.556106 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:38.556075 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.556149 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:38.556103 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:39.515642 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:39.515604 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" event={"ID":"9ee63d99-1900-45ed-a48d-64d2734fef42","Type":"ContainerStarted","Data":"d6861a2372c42a43446c9d5b7ad897bb168a9ab79a0fced5511cf3cd56fff24f"} Nov 02 15:51:39.516075 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:39.515655 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" event={"ID":"9ee63d99-1900-45ed-a48d-64d2734fef42","Type":"ContainerStarted","Data":"a755c15f9d075cdc795d06d6f22afbb1ae62e166aedb1be716a02d4844fe8017"} Nov 02 15:51:39.741201 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:39.741150 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" podStartSLOduration=2.741136123 podStartE2EDuration="2.741136123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:51:39.694469182 +0000 UTC m=+3201.897347254" watchObservedRunningTime="2026-11-02 15:51:39.741136123 +0000 UTC m=+3201.944014183" Nov 02 15:51:40.123733 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:40.123697 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:40.125178 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:40.125144 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wmnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 15:51:40.125314 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:40.125219 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" podUID="9ee63d99-1900-45ed-a48d-64d2734fef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 15:51:41.123586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:41.123554 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wmnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 15:51:41.123988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:41.123603 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" podUID="9ee63d99-1900-45ed-a48d-64d2734fef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 15:51:42.123973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:42.123934 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wmnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 15:51:42.124398 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:42.124013 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" podUID="9ee63d99-1900-45ed-a48d-64d2734fef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 15:51:43.123988 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.123951 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wmnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 15:51:43.124404 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.124030 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" podUID="9ee63d99-1900-45ed-a48d-64d2734fef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 15:51:43.694019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.693675 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp"] Nov 02 15:51:43.699906 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.699840 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:43.745172 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.745129 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ed2a2f-7537-4808-a384-31487155491b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zbphp\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:43.745521 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.745488 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw4l6\" (UniqueName: \"kubernetes.io/projected/61ed2a2f-7537-4808-a384-31487155491b-kube-api-access-qw4l6\") pod \"maas-api-revoke-sub-2e1f5b5f-zbphp\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:43.745644 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.745574 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ed2a2f-7537-4808-a384-31487155491b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zbphp\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:43.746526 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.746506 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp"] Nov 02 15:51:43.750147 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.750125 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:51:43.796257 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.796220 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:51:43.796257 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.796267 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:51:43.796511 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.796412 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:51:43.846934 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.846901 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ed2a2f-7537-4808-a384-31487155491b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zbphp\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:43.847094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.847021 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ed2a2f-7537-4808-a384-31487155491b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zbphp\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:43.847094 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.847060 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw4l6\" (UniqueName: \"kubernetes.io/projected/61ed2a2f-7537-4808-a384-31487155491b-kube-api-access-qw4l6\") pod \"maas-api-revoke-sub-2e1f5b5f-zbphp\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:43.847641 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.847615 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ed2a2f-7537-4808-a384-31487155491b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zbphp\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:43.849372 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.849356 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ed2a2f-7537-4808-a384-31487155491b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-zbphp\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:43.904360 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:43.904333 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw4l6\" (UniqueName: \"kubernetes.io/projected/61ed2a2f-7537-4808-a384-31487155491b-kube-api-access-qw4l6\") pod \"maas-api-revoke-sub-2e1f5b5f-zbphp\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:44.014252 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:44.014177 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:44.123708 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:44.123674 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wmnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 15:51:44.123878 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:44.123738 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" podUID="9ee63d99-1900-45ed-a48d-64d2734fef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 15:51:44.395406 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:44.395331 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp"] Nov 02 15:51:44.398580 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:51:44.398550 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ed2a2f_7537_4808_a384_31487155491b.slice/crio-789b3e40b837f468e8086f25baaf9d872383846de33eaf94e151713214f4e6bf WatchSource:0}: Error finding container 789b3e40b837f468e8086f25baaf9d872383846de33eaf94e151713214f4e6bf: Status 404 returned error can't find the container with id 789b3e40b837f468e8086f25baaf9d872383846de33eaf94e151713214f4e6bf Nov 02 15:51:44.543261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:44.543217 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" event={"ID":"61ed2a2f-7537-4808-a384-31487155491b","Type":"ContainerStarted","Data":"789b3e40b837f468e8086f25baaf9d872383846de33eaf94e151713214f4e6bf"} Nov 02 15:51:45.123289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:45.123255 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-wmnbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 15:51:45.123454 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:45.123304 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" podUID="9ee63d99-1900-45ed-a48d-64d2734fef42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 15:51:45.551834 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:45.551733 5476 generic.go:356] "Generic (PLEG): container finished" podID="61ed2a2f-7537-4808-a384-31487155491b" containerID="d947cf1d8f1457b58e553322a7965667f54bdd88da6bab880fce13608756fcbc" exitCode=0 Nov 02 15:51:45.551834 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:45.551816 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" event={"ID":"61ed2a2f-7537-4808-a384-31487155491b","Type":"ContainerDied","Data":"d947cf1d8f1457b58e553322a7965667f54bdd88da6bab880fce13608756fcbc"} Nov 02 15:51:46.707899 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.707871 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:46.778430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.778401 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ed2a2f-7537-4808-a384-31487155491b-kube-api-access-qw4l6\" (UniqueName: \"kubernetes.io/projected/61ed2a2f-7537-4808-a384-31487155491b-kube-api-access-qw4l6\") pod \"61ed2a2f-7537-4808-a384-31487155491b\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " Nov 02 15:51:46.778581 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.778449 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61ed2a2f-7537-4808-a384-31487155491b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ed2a2f-7537-4808-a384-31487155491b-maas-api-xks-ca\") pod \"61ed2a2f-7537-4808-a384-31487155491b\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " Nov 02 15:51:46.778581 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.778536 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ed2a2f-7537-4808-a384-31487155491b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ed2a2f-7537-4808-a384-31487155491b-maas-api-openshift-ca\") pod \"61ed2a2f-7537-4808-a384-31487155491b\" (UID: \"61ed2a2f-7537-4808-a384-31487155491b\") " Nov 02 15:51:46.778946 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.778905 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ed2a2f-7537-4808-a384-31487155491b-maas-api-openshift-ca" pod "61ed2a2f-7537-4808-a384-31487155491b" (UID: "61ed2a2f-7537-4808-a384-31487155491b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:51:46.780859 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.780831 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ed2a2f-7537-4808-a384-31487155491b-maas-api-xks-ca" pod "61ed2a2f-7537-4808-a384-31487155491b" (UID: "61ed2a2f-7537-4808-a384-31487155491b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:51:46.781223 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.781201 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ed2a2f-7537-4808-a384-31487155491b-kube-api-access-qw4l6" pod "61ed2a2f-7537-4808-a384-31487155491b" (UID: "61ed2a2f-7537-4808-a384-31487155491b"). InnerVolumeSpecName "kube-api-access-qw4l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:51:46.822906 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.822836 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:46.823297 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.823264 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:46.824124 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.824100 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-wmnbp" Nov 02 15:51:46.879838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.879628 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw4l6\" (UniqueName: \"kubernetes.io/projected/61ed2a2f-7537-4808-a384-31487155491b-kube-api-access-qw4l6\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:51:46.879838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.879658 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61ed2a2f-7537-4808-a384-31487155491b-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:51:46.879838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:46.879689 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61ed2a2f-7537-4808-a384-31487155491b-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:51:47.568069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:47.568030 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" event={"ID":"61ed2a2f-7537-4808-a384-31487155491b","Type":"ContainerDied","Data":"789b3e40b837f468e8086f25baaf9d872383846de33eaf94e151713214f4e6bf"} Nov 02 15:51:47.568281 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:47.568099 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="789b3e40b837f468e8086f25baaf9d872383846de33eaf94e151713214f4e6bf" Nov 02 15:51:47.568281 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:47.568047 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp" Nov 02 15:51:50.145551 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:51:50.145515 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:51:56.042416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.042378 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q"] Nov 02 15:51:56.042888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.042810 5476 cpu_manager.go:414] "containerMap: removing container" podUID="61ed2a2f-7537-4808-a384-31487155491b" containerName="revoke-keys" Nov 02 15:51:56.042888 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.042822 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ed2a2f-7537-4808-a384-31487155491b" containerName="revoke-keys" Nov 02 15:51:56.042965 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.042902 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ed2a2f-7537-4808-a384-31487155491b" containerName="revoke-keys" Nov 02 15:51:56.046126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.046110 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:56.141326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.141279 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:51:56.141519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.141437 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:51:56.141564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.141512 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:51:56.141564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.141540 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:51:56.143954 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.143929 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q"] Nov 02 15:51:56.174457 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.174422 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2tb8\" (UniqueName: \"kubernetes.io/projected/eb85ba13-2267-458c-9865-bf3a57841266-kube-api-access-g2tb8\") pod \"maas-api-revoke-sub-ec843018-sg44q\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:56.174636 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.174491 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-sg44q\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:56.174636 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.174590 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-sg44q\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:56.275378 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.275341 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-sg44q\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:56.275560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.275424 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2tb8\" (UniqueName: \"kubernetes.io/projected/eb85ba13-2267-458c-9865-bf3a57841266-kube-api-access-g2tb8\") pod \"maas-api-revoke-sub-ec843018-sg44q\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:56.275560 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.275468 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-sg44q\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:56.276153 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.276130 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-sg44q\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:56.277779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.277756 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-sg44q\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:56.333139 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.333069 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2tb8\" (UniqueName: \"kubernetes.io/projected/eb85ba13-2267-458c-9865-bf3a57841266-kube-api-access-g2tb8\") pod \"maas-api-revoke-sub-ec843018-sg44q\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:56.359205 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.359174 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:56.441725 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.441693 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:51:56.728900 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:56.728863 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q"] Nov 02 15:51:56.735321 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:51:56.735287 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb85ba13_2267_458c_9865_bf3a57841266.slice/crio-a1510a94bb56e0c11c210b0d6e54397ba06a8d3dfa59c5a009dd54aab9a30ea1 WatchSource:0}: Error finding container a1510a94bb56e0c11c210b0d6e54397ba06a8d3dfa59c5a009dd54aab9a30ea1: Status 404 returned error can't find the container with id a1510a94bb56e0c11c210b0d6e54397ba06a8d3dfa59c5a009dd54aab9a30ea1 Nov 02 15:51:57.629036 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:57.628975 5476 generic.go:356] "Generic (PLEG): container finished" podID="eb85ba13-2267-458c-9865-bf3a57841266" containerID="309a5b81c5882498d432468963fd7945c52694ed9258d71b829a3edb575097f8" exitCode=0 Nov 02 15:51:57.629425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:57.629048 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" event={"ID":"eb85ba13-2267-458c-9865-bf3a57841266","Type":"ContainerDied","Data":"309a5b81c5882498d432468963fd7945c52694ed9258d71b829a3edb575097f8"} Nov 02 15:51:57.629425 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:57.629093 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" event={"ID":"eb85ba13-2267-458c-9865-bf3a57841266","Type":"ContainerStarted","Data":"a1510a94bb56e0c11c210b0d6e54397ba06a8d3dfa59c5a009dd54aab9a30ea1"} Nov 02 15:51:58.789637 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:58.789601 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:58.906202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:58.906121 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-xks-ca\") pod \"eb85ba13-2267-458c-9865-bf3a57841266\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " Nov 02 15:51:58.906387 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:58.906201 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb85ba13-2267-458c-9865-bf3a57841266-kube-api-access-g2tb8\" (UniqueName: \"kubernetes.io/projected/eb85ba13-2267-458c-9865-bf3a57841266-kube-api-access-g2tb8\") pod \"eb85ba13-2267-458c-9865-bf3a57841266\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " Nov 02 15:51:58.906734 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:58.906689 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-openshift-ca" pod "eb85ba13-2267-458c-9865-bf3a57841266" (UID: "eb85ba13-2267-458c-9865-bf3a57841266"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:51:58.908854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:58.906761 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-openshift-ca\") pod \"eb85ba13-2267-458c-9865-bf3a57841266\" (UID: \"eb85ba13-2267-458c-9865-bf3a57841266\") " Nov 02 15:51:58.908854 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:58.907193 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:51:58.909195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:58.909168 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-xks-ca" pod "eb85ba13-2267-458c-9865-bf3a57841266" (UID: "eb85ba13-2267-458c-9865-bf3a57841266"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:51:58.910504 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:58.910478 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb85ba13-2267-458c-9865-bf3a57841266-kube-api-access-g2tb8" pod "eb85ba13-2267-458c-9865-bf3a57841266" (UID: "eb85ba13-2267-458c-9865-bf3a57841266"). InnerVolumeSpecName "kube-api-access-g2tb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:51:59.008522 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:59.008483 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2tb8\" (UniqueName: \"kubernetes.io/projected/eb85ba13-2267-458c-9865-bf3a57841266-kube-api-access-g2tb8\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:51:59.008522 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:59.008522 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb85ba13-2267-458c-9865-bf3a57841266-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:51:59.644950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:59.644911 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" Nov 02 15:51:59.644950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:59.644942 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q" event={"ID":"eb85ba13-2267-458c-9865-bf3a57841266","Type":"ContainerDied","Data":"a1510a94bb56e0c11c210b0d6e54397ba06a8d3dfa59c5a009dd54aab9a30ea1"} Nov 02 15:51:59.645197 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:51:59.644990 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1510a94bb56e0c11c210b0d6e54397ba06a8d3dfa59c5a009dd54aab9a30ea1" Nov 02 15:52:13.242440 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:13.242399 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:52:19.833445 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:19.833414 5476 scope.go:122] "RemoveContainer" containerID="f879803b9b4e17806763edc057f15dcbbf83ab4ad7ea97eacc3df932ca29a462" Nov 02 15:52:20.149341 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:52:20.149292 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:52:31.279637 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.279556 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2"] Nov 02 15:52:31.280200 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.280179 5476 cpu_manager.go:414] "containerMap: removing container" podUID="eb85ba13-2267-458c-9865-bf3a57841266" containerName="revoke-keys" Nov 02 15:52:31.280262 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.280204 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb85ba13-2267-458c-9865-bf3a57841266" containerName="revoke-keys" Nov 02 15:52:31.280357 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.280346 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb85ba13-2267-458c-9865-bf3a57841266" containerName="revoke-keys" Nov 02 15:52:31.285527 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.285501 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:31.337067 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.337029 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:52:31.337355 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.337037 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:52:31.383383 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.383350 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:52:31.383559 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.383417 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:52:31.385709 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.385685 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2"] Nov 02 15:52:31.424635 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.424610 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qjxq2\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:31.424760 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.424646 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5czgh\" (UniqueName: \"kubernetes.io/projected/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-kube-api-access-5czgh\") pod \"maas-api-revoke-sub-046e1343-qjxq2\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:31.424807 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.424769 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qjxq2\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:31.525619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.525587 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qjxq2\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:31.525797 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.525626 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5czgh\" (UniqueName: \"kubernetes.io/projected/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-kube-api-access-5czgh\") pod \"maas-api-revoke-sub-046e1343-qjxq2\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:31.525797 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.525676 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qjxq2\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:31.526367 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.526341 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-qjxq2\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:31.528041 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.527990 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-qjxq2\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:31.583846 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.583775 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5czgh\" (UniqueName: \"kubernetes.io/projected/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-kube-api-access-5czgh\") pod \"maas-api-revoke-sub-046e1343-qjxq2\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:31.601428 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.601405 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:31.972987 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:31.972954 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2"] Nov 02 15:52:31.978091 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:52:31.978062 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dea9e7_c2ad_4954_a285_e6beba1f4e01.slice/crio-b77a0d84e18bf33bafd77c7fe539ee28af5f04ae7a67f8c9dc3edc326e99ae68 WatchSource:0}: Error finding container b77a0d84e18bf33bafd77c7fe539ee28af5f04ae7a67f8c9dc3edc326e99ae68: Status 404 returned error can't find the container with id b77a0d84e18bf33bafd77c7fe539ee28af5f04ae7a67f8c9dc3edc326e99ae68 Nov 02 15:52:32.855683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:32.855653 5476 generic.go:356] "Generic (PLEG): container finished" podID="b3dea9e7-c2ad-4954-a285-e6beba1f4e01" containerID="8b8b3c42f4b24a3aaeb598d2af85b9032f1836cabca5bc9b85d83ca6ac8432ae" exitCode=0 Nov 02 15:52:32.856098 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:32.855745 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" event={"ID":"b3dea9e7-c2ad-4954-a285-e6beba1f4e01","Type":"ContainerDied","Data":"8b8b3c42f4b24a3aaeb598d2af85b9032f1836cabca5bc9b85d83ca6ac8432ae"} Nov 02 15:52:32.856098 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:32.855800 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" event={"ID":"b3dea9e7-c2ad-4954-a285-e6beba1f4e01","Type":"ContainerStarted","Data":"b77a0d84e18bf33bafd77c7fe539ee28af5f04ae7a67f8c9dc3edc326e99ae68"} Nov 02 15:52:33.989301 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:33.989275 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:34.151195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.151114 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-xks-ca\") pod \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " Nov 02 15:52:34.151195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.151187 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-openshift-ca\") pod \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " Nov 02 15:52:34.151430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.151300 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-kube-api-access-5czgh\" (UniqueName: \"kubernetes.io/projected/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-kube-api-access-5czgh\") pod \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\" (UID: \"b3dea9e7-c2ad-4954-a285-e6beba1f4e01\") " Nov 02 15:52:34.151543 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.151518 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-openshift-ca" pod "b3dea9e7-c2ad-4954-a285-e6beba1f4e01" (UID: "b3dea9e7-c2ad-4954-a285-e6beba1f4e01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:52:34.153414 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.153391 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-xks-ca" pod "b3dea9e7-c2ad-4954-a285-e6beba1f4e01" (UID: "b3dea9e7-c2ad-4954-a285-e6beba1f4e01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:52:34.153532 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.153428 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-kube-api-access-5czgh" pod "b3dea9e7-c2ad-4954-a285-e6beba1f4e01" (UID: "b3dea9e7-c2ad-4954-a285-e6beba1f4e01"). InnerVolumeSpecName "kube-api-access-5czgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:52:34.252686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.252656 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:52:34.252686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.252682 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5czgh\" (UniqueName: \"kubernetes.io/projected/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-kube-api-access-5czgh\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:52:34.252686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.252692 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dea9e7-c2ad-4954-a285-e6beba1f4e01-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:52:34.868444 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.868413 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" event={"ID":"b3dea9e7-c2ad-4954-a285-e6beba1f4e01","Type":"ContainerDied","Data":"b77a0d84e18bf33bafd77c7fe539ee28af5f04ae7a67f8c9dc3edc326e99ae68"} Nov 02 15:52:34.868444 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.868431 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2" Nov 02 15:52:34.868668 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:34.868459 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b77a0d84e18bf33bafd77c7fe539ee28af5f04ae7a67f8c9dc3edc326e99ae68" Nov 02 15:52:42.055071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.055041 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b"] Nov 02 15:52:42.055471 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.055450 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b3dea9e7-c2ad-4954-a285-e6beba1f4e01" containerName="revoke-keys" Nov 02 15:52:42.055471 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.055462 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dea9e7-c2ad-4954-a285-e6beba1f4e01" containerName="revoke-keys" Nov 02 15:52:42.055550 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.055537 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dea9e7-c2ad-4954-a285-e6beba1f4e01" containerName="revoke-keys" Nov 02 15:52:42.059070 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.059053 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:42.108587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.108554 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:52:42.154436 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.154394 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:52:42.154436 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.154394 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:52:42.154436 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.154412 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:52:42.156880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.156862 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b"] Nov 02 15:52:42.228257 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.228234 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5tg\" (UniqueName: \"kubernetes.io/projected/ef1c909e-78a8-48f4-a28b-fbb1870c059b-kube-api-access-8m5tg\") pod \"maas-api-revoke-sub-d3fba056-dv58b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:42.228390 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.228274 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-dv58b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:42.228457 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.228398 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-dv58b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:42.328936 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.328867 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5tg\" (UniqueName: \"kubernetes.io/projected/ef1c909e-78a8-48f4-a28b-fbb1870c059b-kube-api-access-8m5tg\") pod \"maas-api-revoke-sub-d3fba056-dv58b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:42.328936 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.328922 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-dv58b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:42.329151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.328982 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-dv58b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:42.329561 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.329542 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-dv58b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:42.331175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.331154 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-dv58b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:42.386901 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.386874 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5tg\" (UniqueName: \"kubernetes.io/projected/ef1c909e-78a8-48f4-a28b-fbb1870c059b-kube-api-access-8m5tg\") pod \"maas-api-revoke-sub-d3fba056-dv58b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:42.671730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:42.671698 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:43.043508 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:43.043432 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b"] Nov 02 15:52:43.048918 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:52:43.048888 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1c909e_78a8_48f4_a28b_fbb1870c059b.slice/crio-53cd5084013e2076de81877c16abd09897e5bbf3444c31307ce35412739fa82f WatchSource:0}: Error finding container 53cd5084013e2076de81877c16abd09897e5bbf3444c31307ce35412739fa82f: Status 404 returned error can't find the container with id 53cd5084013e2076de81877c16abd09897e5bbf3444c31307ce35412739fa82f Nov 02 15:52:43.921722 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:43.921689 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" event={"ID":"ef1c909e-78a8-48f4-a28b-fbb1870c059b","Type":"ContainerStarted","Data":"4306674b28bd74adaa96646cd955268dba483ed61b1b3216cab4f8c047e5921e"} Nov 02 15:52:43.922091 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:43.921737 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" event={"ID":"ef1c909e-78a8-48f4-a28b-fbb1870c059b","Type":"ContainerStarted","Data":"53cd5084013e2076de81877c16abd09897e5bbf3444c31307ce35412739fa82f"} Nov 02 15:52:44.018492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:44.018463 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:52:44.030687 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:44.030646 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" podStartSLOduration=2.030633291 podStartE2EDuration="2.030633291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:52:44.030103932 +0000 UTC m=+3266.232981991" watchObservedRunningTime="2026-11-02 15:52:44.030633291 +0000 UTC m=+3266.233511348" Nov 02 15:52:44.929504 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:44.929469 5476 generic.go:356] "Generic (PLEG): container finished" podID="ef1c909e-78a8-48f4-a28b-fbb1870c059b" containerID="4306674b28bd74adaa96646cd955268dba483ed61b1b3216cab4f8c047e5921e" exitCode=0 Nov 02 15:52:44.930165 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:44.929569 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" event={"ID":"ef1c909e-78a8-48f4-a28b-fbb1870c059b","Type":"ContainerDied","Data":"4306674b28bd74adaa96646cd955268dba483ed61b1b3216cab4f8c047e5921e"} Nov 02 15:52:46.325017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.324614 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:46.468526 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.468495 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-xks-ca\") pod \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " Nov 02 15:52:46.468689 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.468564 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1c909e-78a8-48f4-a28b-fbb1870c059b-kube-api-access-8m5tg\" (UniqueName: \"kubernetes.io/projected/ef1c909e-78a8-48f4-a28b-fbb1870c059b-kube-api-access-8m5tg\") pod \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " Nov 02 15:52:46.468774 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.468748 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-openshift-ca\") pod \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\" (UID: \"ef1c909e-78a8-48f4-a28b-fbb1870c059b\") " Nov 02 15:52:46.469090 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.469068 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-openshift-ca" pod "ef1c909e-78a8-48f4-a28b-fbb1870c059b" (UID: "ef1c909e-78a8-48f4-a28b-fbb1870c059b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:52:46.469209 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.469196 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:52:46.470586 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.470555 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1c909e-78a8-48f4-a28b-fbb1870c059b-kube-api-access-8m5tg" pod "ef1c909e-78a8-48f4-a28b-fbb1870c059b" (UID: "ef1c909e-78a8-48f4-a28b-fbb1870c059b"). InnerVolumeSpecName "kube-api-access-8m5tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:52:46.471887 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.471862 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-xks-ca" pod "ef1c909e-78a8-48f4-a28b-fbb1870c059b" (UID: "ef1c909e-78a8-48f4-a28b-fbb1870c059b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:52:46.569726 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.569654 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef1c909e-78a8-48f4-a28b-fbb1870c059b-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:52:46.569726 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.569682 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m5tg\" (UniqueName: \"kubernetes.io/projected/ef1c909e-78a8-48f4-a28b-fbb1870c059b-kube-api-access-8m5tg\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:52:46.964156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.964127 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" Nov 02 15:52:46.964363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.964161 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b" event={"ID":"ef1c909e-78a8-48f4-a28b-fbb1870c059b","Type":"ContainerDied","Data":"53cd5084013e2076de81877c16abd09897e5bbf3444c31307ce35412739fa82f"} Nov 02 15:52:46.964363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:46.964207 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53cd5084013e2076de81877c16abd09897e5bbf3444c31307ce35412739fa82f" Nov 02 15:52:50.145461 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:52:50.145426 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:52:54.782022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:52:54.781972 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:53:06.881577 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:06.881536 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:53:08.050636 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:08.050593 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw"] Nov 02 15:53:08.096990 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:08.096959 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948517fb-ptgnw"] Nov 02 15:53:09.920796 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:09.920761 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1" path="/var/lib/kubelet/pods/ff1b0e7f-4bca-4be2-8f01-02ce88b2b4c1/volumes" Nov 02 15:53:16.858947 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.858914 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76"] Nov 02 15:53:16.859516 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.859320 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ef1c909e-78a8-48f4-a28b-fbb1870c059b" containerName="revoke-keys" Nov 02 15:53:16.859516 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.859332 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1c909e-78a8-48f4-a28b-fbb1870c059b" containerName="revoke-keys" Nov 02 15:53:16.859516 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.859416 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1c909e-78a8-48f4-a28b-fbb1870c059b" containerName="revoke-keys" Nov 02 15:53:16.863754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.863736 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:16.880600 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.880571 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6qg76\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:16.880715 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.880612 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdbz\" (UniqueName: \"kubernetes.io/projected/b1e11201-ff4d-405e-bd68-b4141e2bab8b-kube-api-access-gjdbz\") pod \"maas-api-revoke-sub-ae594979-6qg76\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:16.880715 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.880640 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6qg76\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:16.918325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.918296 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76"] Nov 02 15:53:16.961592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.961551 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:53:16.961592 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.961564 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:53:16.961863 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.961612 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:53:16.961863 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.961619 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:53:16.982162 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.982127 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6qg76\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:16.982336 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.982187 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdbz\" (UniqueName: \"kubernetes.io/projected/b1e11201-ff4d-405e-bd68-b4141e2bab8b-kube-api-access-gjdbz\") pod \"maas-api-revoke-sub-ae594979-6qg76\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:16.982336 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.982219 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6qg76\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:16.982891 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.982865 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-6qg76\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:16.984616 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:16.984590 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-6qg76\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:17.040037 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:17.039989 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdbz\" (UniqueName: \"kubernetes.io/projected/b1e11201-ff4d-405e-bd68-b4141e2bab8b-kube-api-access-gjdbz\") pod \"maas-api-revoke-sub-ae594979-6qg76\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:17.176665 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:17.176567 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:17.549975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:17.549896 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76"] Nov 02 15:53:17.555871 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:53:17.555841 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e11201_ff4d_405e_bd68_b4141e2bab8b.slice/crio-bb2b18032ce3ad47d9471a96b233fae31aeee4a1ad95531a2926a7d6d4d345f7 WatchSource:0}: Error finding container bb2b18032ce3ad47d9471a96b233fae31aeee4a1ad95531a2926a7d6d4d345f7: Status 404 returned error can't find the container with id bb2b18032ce3ad47d9471a96b233fae31aeee4a1ad95531a2926a7d6d4d345f7 Nov 02 15:53:18.145518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:18.145435 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" event={"ID":"b1e11201-ff4d-405e-bd68-b4141e2bab8b","Type":"ContainerStarted","Data":"eb11f0d04ffb2b769d561fb13819e88c3c63991356496f6a75373e1f793d6f5f"} Nov 02 15:53:18.145518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:18.145491 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" event={"ID":"b1e11201-ff4d-405e-bd68-b4141e2bab8b","Type":"ContainerStarted","Data":"bb2b18032ce3ad47d9471a96b233fae31aeee4a1ad95531a2926a7d6d4d345f7"} Nov 02 15:53:18.255518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:18.255457 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" podStartSLOduration=2.255441924 podStartE2EDuration="2.255441924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:53:18.253508916 +0000 UTC m=+3300.456386977" watchObservedRunningTime="2026-11-02 15:53:18.255441924 +0000 UTC m=+3300.458319984" Nov 02 15:53:18.695710 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:18.695683 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:53:18.701969 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:18.701942 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:53:19.153009 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:19.152960 5476 generic.go:356] "Generic (PLEG): container finished" podID="b1e11201-ff4d-405e-bd68-b4141e2bab8b" containerID="eb11f0d04ffb2b769d561fb13819e88c3c63991356496f6a75373e1f793d6f5f" exitCode=0 Nov 02 15:53:19.153404 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:19.153034 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" event={"ID":"b1e11201-ff4d-405e-bd68-b4141e2bab8b","Type":"ContainerDied","Data":"eb11f0d04ffb2b769d561fb13819e88c3c63991356496f6a75373e1f793d6f5f"} Nov 02 15:53:20.146223 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:53:20.146185 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:53:20.303126 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:20.303100 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:20.417154 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:20.417083 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1e11201-ff4d-405e-bd68-b4141e2bab8b-kube-api-access-gjdbz\" (UniqueName: \"kubernetes.io/projected/b1e11201-ff4d-405e-bd68-b4141e2bab8b-kube-api-access-gjdbz\") pod \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " Nov 02 15:53:20.417287 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:20.417154 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-openshift-ca\") pod \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " Nov 02 15:53:20.417287 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:20.417191 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-xks-ca\") pod \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\" (UID: \"b1e11201-ff4d-405e-bd68-b4141e2bab8b\") " Nov 02 15:53:20.418009 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:20.417971 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-openshift-ca" pod "b1e11201-ff4d-405e-bd68-b4141e2bab8b" (UID: "b1e11201-ff4d-405e-bd68-b4141e2bab8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:53:20.419694 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:20.419668 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e11201-ff4d-405e-bd68-b4141e2bab8b-kube-api-access-gjdbz" pod "b1e11201-ff4d-405e-bd68-b4141e2bab8b" (UID: "b1e11201-ff4d-405e-bd68-b4141e2bab8b"). InnerVolumeSpecName "kube-api-access-gjdbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:53:20.419823 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:20.419725 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-xks-ca" pod "b1e11201-ff4d-405e-bd68-b4141e2bab8b" (UID: "b1e11201-ff4d-405e-bd68-b4141e2bab8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:53:20.518551 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:20.518520 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdbz\" (UniqueName: \"kubernetes.io/projected/b1e11201-ff4d-405e-bd68-b4141e2bab8b-kube-api-access-gjdbz\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:53:20.518551 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:20.518547 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:53:20.518727 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:20.518559 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1e11201-ff4d-405e-bd68-b4141e2bab8b-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:53:21.164880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:21.164845 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" event={"ID":"b1e11201-ff4d-405e-bd68-b4141e2bab8b","Type":"ContainerDied","Data":"bb2b18032ce3ad47d9471a96b233fae31aeee4a1ad95531a2926a7d6d4d345f7"} Nov 02 15:53:21.165079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:21.164896 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb2b18032ce3ad47d9471a96b233fae31aeee4a1ad95531a2926a7d6d4d345f7" Nov 02 15:53:21.165079 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:21.164860 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76" Nov 02 15:53:29.658408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:29.658372 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:53:38.865841 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:38.865803 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:53:44.840416 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.840383 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm"] Nov 02 15:53:44.840835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.840803 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b1e11201-ff4d-405e-bd68-b4141e2bab8b" containerName="revoke-keys" Nov 02 15:53:44.840835 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.840815 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1e11201-ff4d-405e-bd68-b4141e2bab8b" containerName="revoke-keys" Nov 02 15:53:44.840967 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.840893 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1e11201-ff4d-405e-bd68-b4141e2bab8b" containerName="revoke-keys" Nov 02 15:53:44.844158 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.844142 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:44.893326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.893284 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:53:44.928428 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.928393 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dc5cad-967b-48f2-8488-655503afa259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pcdwm\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:44.928593 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.928521 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dc5cad-967b-48f2-8488-655503afa259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pcdwm\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:44.928593 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.928544 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8fx6\" (UniqueName: \"kubernetes.io/projected/71dc5cad-967b-48f2-8488-655503afa259-kube-api-access-c8fx6\") pod \"maas-api-revoke-sub-a4aef7d7-pcdwm\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:44.939530 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.939501 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:53:44.939759 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.939741 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:53:44.939833 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.939818 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:53:44.943181 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:44.943158 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm"] Nov 02 15:53:45.030093 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:45.030060 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dc5cad-967b-48f2-8488-655503afa259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pcdwm\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:45.030276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:45.030101 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8fx6\" (UniqueName: \"kubernetes.io/projected/71dc5cad-967b-48f2-8488-655503afa259-kube-api-access-c8fx6\") pod \"maas-api-revoke-sub-a4aef7d7-pcdwm\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:45.030276 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:45.030180 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dc5cad-967b-48f2-8488-655503afa259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pcdwm\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:45.030744 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:45.030720 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dc5cad-967b-48f2-8488-655503afa259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pcdwm\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:45.032546 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:45.032525 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dc5cad-967b-48f2-8488-655503afa259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-pcdwm\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:45.087857 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:45.087826 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8fx6\" (UniqueName: \"kubernetes.io/projected/71dc5cad-967b-48f2-8488-655503afa259-kube-api-access-c8fx6\") pod \"maas-api-revoke-sub-a4aef7d7-pcdwm\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:45.156839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:45.156810 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:45.530383 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:45.530299 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm"] Nov 02 15:53:45.536293 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:53:45.536266 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71dc5cad_967b_48f2_8488_655503afa259.slice/crio-8ab5e08c470d1ced6b620e0ad7a179017a377f6cd0932cc162ccfdb9670ed795 WatchSource:0}: Error finding container 8ab5e08c470d1ced6b620e0ad7a179017a377f6cd0932cc162ccfdb9670ed795: Status 404 returned error can't find the container with id 8ab5e08c470d1ced6b620e0ad7a179017a377f6cd0932cc162ccfdb9670ed795 Nov 02 15:53:46.311571 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:46.311535 5476 generic.go:356] "Generic (PLEG): container finished" podID="71dc5cad-967b-48f2-8488-655503afa259" containerID="b4f293d6015e99de659511be643306d3dc27c33c7c2a6268534868aa810d282d" exitCode=0 Nov 02 15:53:46.311976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:46.311627 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" event={"ID":"71dc5cad-967b-48f2-8488-655503afa259","Type":"ContainerDied","Data":"b4f293d6015e99de659511be643306d3dc27c33c7c2a6268534868aa810d282d"} Nov 02 15:53:46.311976 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:46.311684 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" event={"ID":"71dc5cad-967b-48f2-8488-655503afa259","Type":"ContainerStarted","Data":"8ab5e08c470d1ced6b620e0ad7a179017a377f6cd0932cc162ccfdb9670ed795"} Nov 02 15:53:47.475437 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:47.475406 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:47.556159 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:47.556111 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71dc5cad-967b-48f2-8488-655503afa259-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dc5cad-967b-48f2-8488-655503afa259-maas-api-openshift-ca\") pod \"71dc5cad-967b-48f2-8488-655503afa259\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " Nov 02 15:53:47.556348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:47.556244 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71dc5cad-967b-48f2-8488-655503afa259-kube-api-access-c8fx6\" (UniqueName: \"kubernetes.io/projected/71dc5cad-967b-48f2-8488-655503afa259-kube-api-access-c8fx6\") pod \"71dc5cad-967b-48f2-8488-655503afa259\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " Nov 02 15:53:47.556348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:47.556304 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71dc5cad-967b-48f2-8488-655503afa259-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dc5cad-967b-48f2-8488-655503afa259-maas-api-xks-ca\") pod \"71dc5cad-967b-48f2-8488-655503afa259\" (UID: \"71dc5cad-967b-48f2-8488-655503afa259\") " Nov 02 15:53:47.556552 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:47.556525 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71dc5cad-967b-48f2-8488-655503afa259-maas-api-openshift-ca" pod "71dc5cad-967b-48f2-8488-655503afa259" (UID: "71dc5cad-967b-48f2-8488-655503afa259"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:53:47.556674 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:47.556649 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71dc5cad-967b-48f2-8488-655503afa259-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:53:47.558371 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:47.558340 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71dc5cad-967b-48f2-8488-655503afa259-maas-api-xks-ca" pod "71dc5cad-967b-48f2-8488-655503afa259" (UID: "71dc5cad-967b-48f2-8488-655503afa259"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:53:47.558513 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:47.558493 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dc5cad-967b-48f2-8488-655503afa259-kube-api-access-c8fx6" pod "71dc5cad-967b-48f2-8488-655503afa259" (UID: "71dc5cad-967b-48f2-8488-655503afa259"). InnerVolumeSpecName "kube-api-access-c8fx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:53:47.657390 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:47.657357 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8fx6\" (UniqueName: \"kubernetes.io/projected/71dc5cad-967b-48f2-8488-655503afa259-kube-api-access-c8fx6\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:53:47.657390 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:47.657385 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71dc5cad-967b-48f2-8488-655503afa259-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:53:48.324609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:48.324509 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" event={"ID":"71dc5cad-967b-48f2-8488-655503afa259","Type":"ContainerDied","Data":"8ab5e08c470d1ced6b620e0ad7a179017a377f6cd0932cc162ccfdb9670ed795"} Nov 02 15:53:48.324609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:48.324536 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm" Nov 02 15:53:48.324609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:48.324567 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ab5e08c470d1ced6b620e0ad7a179017a377f6cd0932cc162ccfdb9670ed795" Nov 02 15:53:50.145501 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:53:50.145457 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:53:57.661354 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:53:57.661277 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:54:09.454361 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:09.454321 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:54:15.463549 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.463510 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s"] Nov 02 15:54:15.464043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.463937 5476 cpu_manager.go:414] "containerMap: removing container" podUID="71dc5cad-967b-48f2-8488-655503afa259" containerName="revoke-keys" Nov 02 15:54:15.464043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.463949 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71dc5cad-967b-48f2-8488-655503afa259" containerName="revoke-keys" Nov 02 15:54:15.464043 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.464041 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="71dc5cad-967b-48f2-8488-655503afa259" containerName="revoke-keys" Nov 02 15:54:15.467503 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.467482 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:15.516483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.516451 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:54:15.516483 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.516458 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:54:15.537186 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.537157 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dsfr\" (UniqueName: \"kubernetes.io/projected/d42d847d-7962-4dff-b754-f4e1368d8070-kube-api-access-8dsfr\") pod \"maas-api-revoke-sub-79805ca9-94m2s\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:15.537343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.537284 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-94m2s\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:15.537343 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.537334 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-94m2s\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:15.562638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.562611 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:54:15.562768 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.562647 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:54:15.565136 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.565118 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s"] Nov 02 15:54:15.638128 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.638097 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-94m2s\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:15.638261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.638157 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-94m2s\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:15.638261 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.638223 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dsfr\" (UniqueName: \"kubernetes.io/projected/d42d847d-7962-4dff-b754-f4e1368d8070-kube-api-access-8dsfr\") pod \"maas-api-revoke-sub-79805ca9-94m2s\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:15.638870 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.638848 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-94m2s\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:15.640408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.640388 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-94m2s\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:15.695637 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.695603 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dsfr\" (UniqueName: \"kubernetes.io/projected/d42d847d-7962-4dff-b754-f4e1368d8070-kube-api-access-8dsfr\") pod \"maas-api-revoke-sub-79805ca9-94m2s\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:15.783053 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:15.782979 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:16.155027 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:16.154987 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s"] Nov 02 15:54:16.160847 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:54:16.160811 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42d847d_7962_4dff_b754_f4e1368d8070.slice/crio-be88e033dc934336612136215a1a470f1d7a25eb6c387dd0b04d0a73f3064cd4 WatchSource:0}: Error finding container be88e033dc934336612136215a1a470f1d7a25eb6c387dd0b04d0a73f3064cd4: Status 404 returned error can't find the container with id be88e033dc934336612136215a1a470f1d7a25eb6c387dd0b04d0a73f3064cd4 Nov 02 15:54:16.504251 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:16.504215 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" event={"ID":"d42d847d-7962-4dff-b754-f4e1368d8070","Type":"ContainerStarted","Data":"587afe38dc115f3128ea9d99db00011bc33f560fb2bd8ec690c345cc0ccd8649"} Nov 02 15:54:16.504634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:16.504274 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" event={"ID":"d42d847d-7962-4dff-b754-f4e1368d8070","Type":"ContainerStarted","Data":"be88e033dc934336612136215a1a470f1d7a25eb6c387dd0b04d0a73f3064cd4"} Nov 02 15:54:16.615037 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:16.614979 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" podStartSLOduration=1.614966186 podStartE2EDuration="1.614966186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:54:16.612737637 +0000 UTC m=+3358.815615700" watchObservedRunningTime="2026-11-02 15:54:16.614966186 +0000 UTC m=+3358.817844266" Nov 02 15:54:17.047239 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:17.047206 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj"] Nov 02 15:54:17.093133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:17.093110 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa11236-g22tj"] Nov 02 15:54:17.510926 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:17.510891 5476 generic.go:356] "Generic (PLEG): container finished" podID="d42d847d-7962-4dff-b754-f4e1368d8070" containerID="587afe38dc115f3128ea9d99db00011bc33f560fb2bd8ec690c345cc0ccd8649" exitCode=0 Nov 02 15:54:17.511369 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:17.510935 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" event={"ID":"d42d847d-7962-4dff-b754-f4e1368d8070","Type":"ContainerDied","Data":"587afe38dc115f3128ea9d99db00011bc33f560fb2bd8ec690c345cc0ccd8649"} Nov 02 15:54:17.921522 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:17.921485 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ea013a-bfb2-46c4-a837-99d864255101" path="/var/lib/kubelet/pods/f2ea013a-bfb2-46c4-a837-99d864255101/volumes" Nov 02 15:54:18.650929 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:18.650902 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:18.767468 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:18.767440 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42d847d-7962-4dff-b754-f4e1368d8070-kube-api-access-8dsfr\" (UniqueName: \"kubernetes.io/projected/d42d847d-7962-4dff-b754-f4e1368d8070-kube-api-access-8dsfr\") pod \"d42d847d-7962-4dff-b754-f4e1368d8070\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " Nov 02 15:54:18.767627 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:18.767485 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-xks-ca\") pod \"d42d847d-7962-4dff-b754-f4e1368d8070\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " Nov 02 15:54:18.767627 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:18.767550 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-openshift-ca\") pod \"d42d847d-7962-4dff-b754-f4e1368d8070\" (UID: \"d42d847d-7962-4dff-b754-f4e1368d8070\") " Nov 02 15:54:18.768034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:18.767950 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-openshift-ca" pod "d42d847d-7962-4dff-b754-f4e1368d8070" (UID: "d42d847d-7962-4dff-b754-f4e1368d8070"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:54:18.769638 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:18.769609 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42d847d-7962-4dff-b754-f4e1368d8070-kube-api-access-8dsfr" pod "d42d847d-7962-4dff-b754-f4e1368d8070" (UID: "d42d847d-7962-4dff-b754-f4e1368d8070"). InnerVolumeSpecName "kube-api-access-8dsfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:54:18.769822 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:18.769804 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-xks-ca" pod "d42d847d-7962-4dff-b754-f4e1368d8070" (UID: "d42d847d-7962-4dff-b754-f4e1368d8070"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:54:18.868730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:18.868703 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:54:18.868730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:18.868727 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d42d847d-7962-4dff-b754-f4e1368d8070-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:54:18.868890 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:18.868737 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dsfr\" (UniqueName: \"kubernetes.io/projected/d42d847d-7962-4dff-b754-f4e1368d8070-kube-api-access-8dsfr\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:54:19.524202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:19.524163 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" event={"ID":"d42d847d-7962-4dff-b754-f4e1368d8070","Type":"ContainerDied","Data":"be88e033dc934336612136215a1a470f1d7a25eb6c387dd0b04d0a73f3064cd4"} Nov 02 15:54:19.524202 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:19.524210 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be88e033dc934336612136215a1a470f1d7a25eb6c387dd0b04d0a73f3064cd4" Nov 02 15:54:19.524474 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:19.524219 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s" Nov 02 15:54:19.865199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:19.865165 5476 scope.go:122] "RemoveContainer" containerID="ae2653e19aadfdd29660395fc9cf59e482039f0c197ebb72f9ab431afa0b23fd" Nov 02 15:54:20.145483 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:54:20.145400 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:54:29.943300 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:29.943266 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:54:39.247793 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.247757 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp"] Nov 02 15:54:39.248219 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.248164 5476 cpu_manager.go:414] "containerMap: removing container" podUID="d42d847d-7962-4dff-b754-f4e1368d8070" containerName="revoke-keys" Nov 02 15:54:39.248219 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.248175 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42d847d-7962-4dff-b754-f4e1368d8070" containerName="revoke-keys" Nov 02 15:54:39.248295 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.248247 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42d847d-7962-4dff-b754-f4e1368d8070" containerName="revoke-keys" Nov 02 15:54:39.252771 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.252748 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:39.304564 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.304536 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:54:39.308144 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.308114 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp"] Nov 02 15:54:39.351271 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.351240 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:54:39.351271 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.351265 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:54:39.351465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.351351 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:54:39.404272 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.404246 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-24nfp\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:39.404409 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.404320 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-24nfp\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:39.404487 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.404432 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf56d\" (UniqueName: \"kubernetes.io/projected/b2507e56-412b-429e-8703-bc465bdcf325-kube-api-access-hf56d\") pod \"maas-api-revoke-sub-0b7e8edb-24nfp\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:39.505872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.505809 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-24nfp\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:39.505872 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.505850 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf56d\" (UniqueName: \"kubernetes.io/projected/b2507e56-412b-429e-8703-bc465bdcf325-kube-api-access-hf56d\") pod \"maas-api-revoke-sub-0b7e8edb-24nfp\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:39.506049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.505901 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-24nfp\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:39.506611 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.506589 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-24nfp\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:39.508137 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.508116 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-24nfp\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:39.563670 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.563644 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf56d\" (UniqueName: \"kubernetes.io/projected/b2507e56-412b-429e-8703-bc465bdcf325-kube-api-access-hf56d\") pod \"maas-api-revoke-sub-0b7e8edb-24nfp\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:39.565470 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.565445 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:39.936839 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:39.936805 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp"] Nov 02 15:54:39.941957 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:54:39.941930 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2507e56_412b_429e_8703_bc465bdcf325.slice/crio-495f10b1638f46028bade1cb8b4ccc549e746ee475597e931d1f1255e09c29ee WatchSource:0}: Error finding container 495f10b1638f46028bade1cb8b4ccc549e746ee475597e931d1f1255e09c29ee: Status 404 returned error can't find the container with id 495f10b1638f46028bade1cb8b4ccc549e746ee475597e931d1f1255e09c29ee Nov 02 15:54:40.693227 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:40.693194 5476 generic.go:356] "Generic (PLEG): container finished" podID="b2507e56-412b-429e-8703-bc465bdcf325" containerID="55504cbef51913245abca164afd6e5ee20e0ff292de9abb319c6d6fb755f8fb8" exitCode=0 Nov 02 15:54:40.693609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:40.693273 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" event={"ID":"b2507e56-412b-429e-8703-bc465bdcf325","Type":"ContainerDied","Data":"55504cbef51913245abca164afd6e5ee20e0ff292de9abb319c6d6fb755f8fb8"} Nov 02 15:54:40.693609 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:40.693324 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" event={"ID":"b2507e56-412b-429e-8703-bc465bdcf325","Type":"ContainerStarted","Data":"495f10b1638f46028bade1cb8b4ccc549e746ee475597e931d1f1255e09c29ee"} Nov 02 15:54:41.831697 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:41.831674 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:41.924637 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:41.924605 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-xks-ca\") pod \"b2507e56-412b-429e-8703-bc465bdcf325\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " Nov 02 15:54:41.927061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:41.927022 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-xks-ca" pod "b2507e56-412b-429e-8703-bc465bdcf325" (UID: "b2507e56-412b-429e-8703-bc465bdcf325"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:54:42.025656 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:42.025577 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2507e56-412b-429e-8703-bc465bdcf325-kube-api-access-hf56d\" (UniqueName: \"kubernetes.io/projected/b2507e56-412b-429e-8703-bc465bdcf325-kube-api-access-hf56d\") pod \"b2507e56-412b-429e-8703-bc465bdcf325\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " Nov 02 15:54:42.025656 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:42.025627 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-openshift-ca\") pod \"b2507e56-412b-429e-8703-bc465bdcf325\" (UID: \"b2507e56-412b-429e-8703-bc465bdcf325\") " Nov 02 15:54:42.026020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:42.025961 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:54:42.026151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:42.026033 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-openshift-ca" pod "b2507e56-412b-429e-8703-bc465bdcf325" (UID: "b2507e56-412b-429e-8703-bc465bdcf325"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:54:42.027800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:42.027777 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2507e56-412b-429e-8703-bc465bdcf325-kube-api-access-hf56d" pod "b2507e56-412b-429e-8703-bc465bdcf325" (UID: "b2507e56-412b-429e-8703-bc465bdcf325"). InnerVolumeSpecName "kube-api-access-hf56d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:54:42.126887 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:42.126857 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf56d\" (UniqueName: \"kubernetes.io/projected/b2507e56-412b-429e-8703-bc465bdcf325-kube-api-access-hf56d\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:54:42.126887 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:42.126883 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2507e56-412b-429e-8703-bc465bdcf325-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:54:42.705767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:42.705724 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" event={"ID":"b2507e56-412b-429e-8703-bc465bdcf325","Type":"ContainerDied","Data":"495f10b1638f46028bade1cb8b4ccc549e746ee475597e931d1f1255e09c29ee"} Nov 02 15:54:42.705767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:42.705752 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp" Nov 02 15:54:42.705767 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:42.705773 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="495f10b1638f46028bade1cb8b4ccc549e746ee475597e931d1f1255e09c29ee" Nov 02 15:54:50.145914 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:54:50.145877 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:54:50.563390 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.563299 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj"] Nov 02 15:54:50.563991 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.563968 5476 cpu_manager.go:414] "containerMap: removing container" podUID="b2507e56-412b-429e-8703-bc465bdcf325" containerName="revoke-keys" Nov 02 15:54:50.563991 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.564010 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2507e56-412b-429e-8703-bc465bdcf325" containerName="revoke-keys" Nov 02 15:54:50.564151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.564140 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2507e56-412b-429e-8703-bc465bdcf325" containerName="revoke-keys" Nov 02 15:54:50.567639 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.567620 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:50.607063 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.607033 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k45sj\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:50.607232 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.607106 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkbhl\" (UniqueName: \"kubernetes.io/projected/57846fe0-53f7-4188-95ec-8b24e53ec25c-kube-api-access-tkbhl\") pod \"maas-api-revoke-sub-3d4156f1-k45sj\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:50.607281 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.607243 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k45sj\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:50.617848 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.617826 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:54:50.663777 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.663746 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:54:50.663952 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.663857 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:54:50.664056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.663982 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:54:50.666476 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.666455 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj"] Nov 02 15:54:50.707990 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.707966 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k45sj\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:50.708127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.708058 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkbhl\" (UniqueName: \"kubernetes.io/projected/57846fe0-53f7-4188-95ec-8b24e53ec25c-kube-api-access-tkbhl\") pod \"maas-api-revoke-sub-3d4156f1-k45sj\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:50.708127 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.708091 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k45sj\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:50.708608 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.708585 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-k45sj\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:50.710327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.710304 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-k45sj\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:50.765880 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.765855 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkbhl\" (UniqueName: \"kubernetes.io/projected/57846fe0-53f7-4188-95ec-8b24e53ec25c-kube-api-access-tkbhl\") pod \"maas-api-revoke-sub-3d4156f1-k45sj\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:50.881209 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:50.881184 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:51.252537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:51.252455 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj"] Nov 02 15:54:51.258070 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:54:51.258042 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57846fe0_53f7_4188_95ec_8b24e53ec25c.slice/crio-a459916931f05e0c72e34089628917524f59f132e45a021260e4eaf7d72382b0 WatchSource:0}: Error finding container a459916931f05e0c72e34089628917524f59f132e45a021260e4eaf7d72382b0: Status 404 returned error can't find the container with id a459916931f05e0c72e34089628917524f59f132e45a021260e4eaf7d72382b0 Nov 02 15:54:51.757073 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:51.757035 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" event={"ID":"57846fe0-53f7-4188-95ec-8b24e53ec25c","Type":"ContainerStarted","Data":"e8218418a18fe57307a67d4dd6ee4f6789796942d325c2ec5846ff3949ee8acc"} Nov 02 15:54:51.757225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:51.757091 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" event={"ID":"57846fe0-53f7-4188-95ec-8b24e53ec25c","Type":"ContainerStarted","Data":"a459916931f05e0c72e34089628917524f59f132e45a021260e4eaf7d72382b0"} Nov 02 15:54:51.866195 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:51.866118 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" podStartSLOduration=1.86610347 podStartE2EDuration="1.86610347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:54:51.86560635 +0000 UTC m=+3394.068484409" watchObservedRunningTime="2026-11-02 15:54:51.86610347 +0000 UTC m=+3394.068981529" Nov 02 15:54:52.050184 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:52.050147 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:54:52.764181 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:52.764147 5476 generic.go:356] "Generic (PLEG): container finished" podID="57846fe0-53f7-4188-95ec-8b24e53ec25c" containerID="e8218418a18fe57307a67d4dd6ee4f6789796942d325c2ec5846ff3949ee8acc" exitCode=0 Nov 02 15:54:52.764587 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:52.764187 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" event={"ID":"57846fe0-53f7-4188-95ec-8b24e53ec25c","Type":"ContainerDied","Data":"e8218418a18fe57307a67d4dd6ee4f6789796942d325c2ec5846ff3949ee8acc"} Nov 02 15:54:54.143788 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.143758 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:54.163973 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.163947 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57846fe0-53f7-4188-95ec-8b24e53ec25c-kube-api-access-tkbhl\" (UniqueName: \"kubernetes.io/projected/57846fe0-53f7-4188-95ec-8b24e53ec25c-kube-api-access-tkbhl\") pod \"57846fe0-53f7-4188-95ec-8b24e53ec25c\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " Nov 02 15:54:54.164124 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.163988 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-openshift-ca\") pod \"57846fe0-53f7-4188-95ec-8b24e53ec25c\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " Nov 02 15:54:54.164229 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.164202 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-xks-ca\") pod \"57846fe0-53f7-4188-95ec-8b24e53ec25c\" (UID: \"57846fe0-53f7-4188-95ec-8b24e53ec25c\") " Nov 02 15:54:54.164308 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.164291 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-openshift-ca" pod "57846fe0-53f7-4188-95ec-8b24e53ec25c" (UID: "57846fe0-53f7-4188-95ec-8b24e53ec25c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:54:54.164681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.164653 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:54:54.166077 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.166053 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57846fe0-53f7-4188-95ec-8b24e53ec25c-kube-api-access-tkbhl" pod "57846fe0-53f7-4188-95ec-8b24e53ec25c" (UID: "57846fe0-53f7-4188-95ec-8b24e53ec25c"). InnerVolumeSpecName "kube-api-access-tkbhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:54:54.166204 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.166184 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-xks-ca" pod "57846fe0-53f7-4188-95ec-8b24e53ec25c" (UID: "57846fe0-53f7-4188-95ec-8b24e53ec25c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:54:54.265751 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.265719 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkbhl\" (UniqueName: \"kubernetes.io/projected/57846fe0-53f7-4188-95ec-8b24e53ec25c-kube-api-access-tkbhl\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:54:54.265751 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.265750 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57846fe0-53f7-4188-95ec-8b24e53ec25c-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:54:54.777246 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.777215 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" Nov 02 15:54:54.777420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.777250 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj" event={"ID":"57846fe0-53f7-4188-95ec-8b24e53ec25c","Type":"ContainerDied","Data":"a459916931f05e0c72e34089628917524f59f132e45a021260e4eaf7d72382b0"} Nov 02 15:54:54.777420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:54:54.777323 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a459916931f05e0c72e34089628917524f59f132e45a021260e4eaf7d72382b0" Nov 02 15:55:04.966956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:04.966916 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:55:11.693698 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.693663 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj"] Nov 02 15:55:11.739649 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.739615 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-k45sj"] Nov 02 15:55:11.818661 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.818631 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd"] Nov 02 15:55:11.819061 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.819046 5476 cpu_manager.go:414] "containerMap: removing container" podUID="57846fe0-53f7-4188-95ec-8b24e53ec25c" containerName="revoke-keys" Nov 02 15:55:11.819117 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.819062 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57846fe0-53f7-4188-95ec-8b24e53ec25c" containerName="revoke-keys" Nov 02 15:55:11.819156 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.819140 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="57846fe0-53f7-4188-95ec-8b24e53ec25c" containerName="revoke-keys" Nov 02 15:55:11.822653 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.822636 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:11.874716 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.874693 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:55:11.874864 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.874740 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:55:11.874864 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.874767 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:55:11.874864 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.874863 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:55:11.881303 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.881280 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd"] Nov 02 15:55:11.890208 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.890188 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ftjfd\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:11.890330 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.890226 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc9th\" (UniqueName: \"kubernetes.io/projected/c3cc76da-29f3-4450-b4ba-676978e7d10b-kube-api-access-fc9th\") pod \"maas-api-revoke-sub-3d4156f1-ftjfd\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:11.890330 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.890254 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ftjfd\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:11.920542 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.920515 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57846fe0-53f7-4188-95ec-8b24e53ec25c" path="/var/lib/kubelet/pods/57846fe0-53f7-4188-95ec-8b24e53ec25c/volumes" Nov 02 15:55:11.990955 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.990865 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ftjfd\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:11.990955 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.990930 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc9th\" (UniqueName: \"kubernetes.io/projected/c3cc76da-29f3-4450-b4ba-676978e7d10b-kube-api-access-fc9th\") pod \"maas-api-revoke-sub-3d4156f1-ftjfd\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:11.991235 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.991111 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ftjfd\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:11.991671 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.991652 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ftjfd\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:11.993430 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:11.993410 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ftjfd\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:12.048327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:12.048295 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4"] Nov 02 15:55:12.051133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:12.051110 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc9th\" (UniqueName: \"kubernetes.io/projected/c3cc76da-29f3-4450-b4ba-676978e7d10b-kube-api-access-fc9th\") pod \"maas-api-revoke-sub-3d4156f1-ftjfd\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:12.052235 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:12.052216 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-bvbl4"] Nov 02 15:55:12.135408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:12.135377 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:12.505975 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:12.505927 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd"] Nov 02 15:55:12.511521 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:55:12.511493 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cc76da_29f3_4450_b4ba_676978e7d10b.slice/crio-3169c2ced5c66d3c688a17455ddb35b27d740ee56ba7d938714ba22e7c8e41f0 WatchSource:0}: Error finding container 3169c2ced5c66d3c688a17455ddb35b27d740ee56ba7d938714ba22e7c8e41f0: Status 404 returned error can't find the container with id 3169c2ced5c66d3c688a17455ddb35b27d740ee56ba7d938714ba22e7c8e41f0 Nov 02 15:55:12.887754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:12.887715 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" event={"ID":"c3cc76da-29f3-4450-b4ba-676978e7d10b","Type":"ContainerStarted","Data":"4be58ce97f86e15f2f807131f412a655b385b238de92e291c507ed53f0ba1e5c"} Nov 02 15:55:12.888165 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:12.887779 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" event={"ID":"c3cc76da-29f3-4450-b4ba-676978e7d10b","Type":"ContainerStarted","Data":"3169c2ced5c66d3c688a17455ddb35b27d740ee56ba7d938714ba22e7c8e41f0"} Nov 02 15:55:12.997667 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:12.997615 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" podStartSLOduration=1.997602175 podStartE2EDuration="1.997602175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:55:12.99582542 +0000 UTC m=+3415.198703480" watchObservedRunningTime="2026-11-02 15:55:12.997602175 +0000 UTC m=+3415.200480234" Nov 02 15:55:13.895706 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:13.895670 5476 generic.go:356] "Generic (PLEG): container finished" podID="c3cc76da-29f3-4450-b4ba-676978e7d10b" containerID="4be58ce97f86e15f2f807131f412a655b385b238de92e291c507ed53f0ba1e5c" exitCode=0 Nov 02 15:55:13.896119 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:13.895715 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" event={"ID":"c3cc76da-29f3-4450-b4ba-676978e7d10b","Type":"ContainerDied","Data":"4be58ce97f86e15f2f807131f412a655b385b238de92e291c507ed53f0ba1e5c"} Nov 02 15:55:13.921289 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:13.921259 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f41fe48-373c-4a7d-bf00-aed6496ac6a2" path="/var/lib/kubelet/pods/7f41fe48-373c-4a7d-bf00-aed6496ac6a2/volumes" Nov 02 15:55:15.059071 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.059045 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:15.120434 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.120403 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-xks-ca\") pod \"c3cc76da-29f3-4450-b4ba-676978e7d10b\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " Nov 02 15:55:15.120601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.120475 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-openshift-ca\") pod \"c3cc76da-29f3-4450-b4ba-676978e7d10b\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " Nov 02 15:55:15.120601 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.120554 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3cc76da-29f3-4450-b4ba-676978e7d10b-kube-api-access-fc9th\" (UniqueName: \"kubernetes.io/projected/c3cc76da-29f3-4450-b4ba-676978e7d10b-kube-api-access-fc9th\") pod \"c3cc76da-29f3-4450-b4ba-676978e7d10b\" (UID: \"c3cc76da-29f3-4450-b4ba-676978e7d10b\") " Nov 02 15:55:15.120858 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.120831 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-openshift-ca" pod "c3cc76da-29f3-4450-b4ba-676978e7d10b" (UID: "c3cc76da-29f3-4450-b4ba-676978e7d10b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:55:15.122546 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.122521 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-xks-ca" pod "c3cc76da-29f3-4450-b4ba-676978e7d10b" (UID: "c3cc76da-29f3-4450-b4ba-676978e7d10b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:55:15.122698 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.122677 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cc76da-29f3-4450-b4ba-676978e7d10b-kube-api-access-fc9th" pod "c3cc76da-29f3-4450-b4ba-676978e7d10b" (UID: "c3cc76da-29f3-4450-b4ba-676978e7d10b"). InnerVolumeSpecName "kube-api-access-fc9th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:55:15.221279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.221197 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:55:15.221279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.221228 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc9th\" (UniqueName: \"kubernetes.io/projected/c3cc76da-29f3-4450-b4ba-676978e7d10b-kube-api-access-fc9th\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:55:15.221279 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.221238 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3cc76da-29f3-4450-b4ba-676978e7d10b-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:55:15.907819 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.907788 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" Nov 02 15:55:15.908026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.907794 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd" event={"ID":"c3cc76da-29f3-4450-b4ba-676978e7d10b","Type":"ContainerDied","Data":"3169c2ced5c66d3c688a17455ddb35b27d740ee56ba7d938714ba22e7c8e41f0"} Nov 02 15:55:15.908026 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:15.907947 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3169c2ced5c66d3c688a17455ddb35b27d740ee56ba7d938714ba22e7c8e41f0" Nov 02 15:55:19.887328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:19.887299 5476 scope.go:122] "RemoveContainer" containerID="3fd238b318fc07f562632037657d1db5225577b4182c0e5166f7333ab04e3dee" Nov 02 15:55:19.898898 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:19.898880 5476 scope.go:122] "RemoveContainer" containerID="b05e8ec22461f77b5ff4e11712cb0121478e95b0b875f9e3bbbcb50f9f76c031" Nov 02 15:55:20.145150 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:55:20.145067 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:55:39.080686 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:39.080607 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:55:49.474871 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.474836 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5"] Nov 02 15:55:49.475526 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.475504 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c3cc76da-29f3-4450-b4ba-676978e7d10b" containerName="revoke-keys" Nov 02 15:55:49.475631 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.475527 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3cc76da-29f3-4450-b4ba-676978e7d10b" containerName="revoke-keys" Nov 02 15:55:49.475631 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.475626 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3cc76da-29f3-4450-b4ba-676978e7d10b" containerName="revoke-keys" Nov 02 15:55:49.478779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.478755 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:49.511523 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.511487 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-btlq5\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:49.511703 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.511606 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-btlq5\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:49.511703 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.511638 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qtw\" (UniqueName: \"kubernetes.io/projected/08f73102-ec90-4c11-94ab-ff616719bc02-kube-api-access-k6qtw\") pod \"maas-api-revoke-sub-e2fc9c5b-btlq5\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:49.558320 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.558284 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:55:49.558320 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.558284 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:55:49.558544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.558413 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:55:49.558544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.558458 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:55:49.576433 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.576395 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5"] Nov 02 15:55:49.612373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.612338 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-btlq5\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:49.612582 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.612412 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-btlq5\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:49.612582 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.612439 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qtw\" (UniqueName: \"kubernetes.io/projected/08f73102-ec90-4c11-94ab-ff616719bc02-kube-api-access-k6qtw\") pod \"maas-api-revoke-sub-e2fc9c5b-btlq5\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:49.613194 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.613162 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-btlq5\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:49.614730 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.614711 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-btlq5\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:49.670167 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.670137 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qtw\" (UniqueName: \"kubernetes.io/projected/08f73102-ec90-4c11-94ab-ff616719bc02-kube-api-access-k6qtw\") pod \"maas-api-revoke-sub-e2fc9c5b-btlq5\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:49.791763 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:49.791681 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:50.146598 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:55:50.146555 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:55:50.163918 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:50.163887 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5"] Nov 02 15:55:50.169512 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:55:50.169478 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f73102_ec90_4c11_94ab_ff616719bc02.slice/crio-8edde83fd73bcd3f992e3ee2c93ccdd03dad762f0b83e44ba31c6cf7706800c0 WatchSource:0}: Error finding container 8edde83fd73bcd3f992e3ee2c93ccdd03dad762f0b83e44ba31c6cf7706800c0: Status 404 returned error can't find the container with id 8edde83fd73bcd3f992e3ee2c93ccdd03dad762f0b83e44ba31c6cf7706800c0 Nov 02 15:55:51.121114 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:51.121072 5476 generic.go:356] "Generic (PLEG): container finished" podID="08f73102-ec90-4c11-94ab-ff616719bc02" containerID="1af9207bd6b1015075a60883466675ea167977ca684106d56d60a6db3349bda6" exitCode=0 Nov 02 15:55:51.121544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:51.121157 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" event={"ID":"08f73102-ec90-4c11-94ab-ff616719bc02","Type":"ContainerDied","Data":"1af9207bd6b1015075a60883466675ea167977ca684106d56d60a6db3349bda6"} Nov 02 15:55:51.121544 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:51.121222 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" event={"ID":"08f73102-ec90-4c11-94ab-ff616719bc02","Type":"ContainerStarted","Data":"8edde83fd73bcd3f992e3ee2c93ccdd03dad762f0b83e44ba31c6cf7706800c0"} Nov 02 15:55:52.253411 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:52.253388 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:52.340201 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:52.340175 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-openshift-ca\") pod \"08f73102-ec90-4c11-94ab-ff616719bc02\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " Nov 02 15:55:52.340357 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:52.340215 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-xks-ca\") pod \"08f73102-ec90-4c11-94ab-ff616719bc02\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " Nov 02 15:55:52.340357 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:52.340254 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f73102-ec90-4c11-94ab-ff616719bc02-kube-api-access-k6qtw\" (UniqueName: \"kubernetes.io/projected/08f73102-ec90-4c11-94ab-ff616719bc02-kube-api-access-k6qtw\") pod \"08f73102-ec90-4c11-94ab-ff616719bc02\" (UID: \"08f73102-ec90-4c11-94ab-ff616719bc02\") " Nov 02 15:55:52.340568 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:52.340546 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-openshift-ca" pod "08f73102-ec90-4c11-94ab-ff616719bc02" (UID: "08f73102-ec90-4c11-94ab-ff616719bc02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:55:52.342151 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:52.342127 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-xks-ca" pod "08f73102-ec90-4c11-94ab-ff616719bc02" (UID: "08f73102-ec90-4c11-94ab-ff616719bc02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:55:52.342285 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:52.342230 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f73102-ec90-4c11-94ab-ff616719bc02-kube-api-access-k6qtw" pod "08f73102-ec90-4c11-94ab-ff616719bc02" (UID: "08f73102-ec90-4c11-94ab-ff616719bc02"). InnerVolumeSpecName "kube-api-access-k6qtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:55:52.442123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:52.442032 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:55:52.442123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:52.442074 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08f73102-ec90-4c11-94ab-ff616719bc02-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:55:52.442123 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:52.442089 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6qtw\" (UniqueName: \"kubernetes.io/projected/08f73102-ec90-4c11-94ab-ff616719bc02-kube-api-access-k6qtw\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:55:53.133325 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:53.133297 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" Nov 02 15:55:53.133508 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:53.133290 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5" event={"ID":"08f73102-ec90-4c11-94ab-ff616719bc02","Type":"ContainerDied","Data":"8edde83fd73bcd3f992e3ee2c93ccdd03dad762f0b83e44ba31c6cf7706800c0"} Nov 02 15:55:53.133508 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:55:53.133441 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8edde83fd73bcd3f992e3ee2c93ccdd03dad762f0b83e44ba31c6cf7706800c0" Nov 02 15:56:00.200467 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.200435 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452"] Nov 02 15:56:00.200874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.200860 5476 cpu_manager.go:414] "containerMap: removing container" podUID="08f73102-ec90-4c11-94ab-ff616719bc02" containerName="revoke-keys" Nov 02 15:56:00.200921 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.200876 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f73102-ec90-4c11-94ab-ff616719bc02" containerName="revoke-keys" Nov 02 15:56:00.200972 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.200961 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f73102-ec90-4c11-94ab-ff616719bc02" containerName="revoke-keys" Nov 02 15:56:00.204332 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.204313 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:00.299514 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.299480 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:56:00.299765 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.299745 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:56:00.299842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.299752 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:56:00.299842 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.299753 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:56:00.302234 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.302213 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452"] Nov 02 15:56:00.319113 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.319089 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-b9452\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:00.319253 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.319163 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6ld\" (UniqueName: \"kubernetes.io/projected/c228fa0a-24ca-410e-90ff-3fa2b82000e9-kube-api-access-7t6ld\") pod \"maas-api-revoke-sub-5a29efb3-b9452\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:00.319322 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.319292 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-b9452\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:00.420387 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.420361 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-b9452\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:00.420519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.420411 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6ld\" (UniqueName: \"kubernetes.io/projected/c228fa0a-24ca-410e-90ff-3fa2b82000e9-kube-api-access-7t6ld\") pod \"maas-api-revoke-sub-5a29efb3-b9452\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:00.420519 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.420456 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-b9452\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:00.421069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.421050 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-b9452\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:00.422683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.422664 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-b9452\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:00.477958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.477893 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6ld\" (UniqueName: \"kubernetes.io/projected/c228fa0a-24ca-410e-90ff-3fa2b82000e9-kube-api-access-7t6ld\") pod \"maas-api-revoke-sub-5a29efb3-b9452\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:00.517022 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.516981 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:00.886216 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:00.886182 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452"] Nov 02 15:56:00.891755 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:56:00.891721 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc228fa0a_24ca_410e_90ff_3fa2b82000e9.slice/crio-b09035b03df095691ab9d9fd9909649d0a3150ead413946132772b7208f583fd WatchSource:0}: Error finding container b09035b03df095691ab9d9fd9909649d0a3150ead413946132772b7208f583fd: Status 404 returned error can't find the container with id b09035b03df095691ab9d9fd9909649d0a3150ead413946132772b7208f583fd Nov 02 15:56:01.178745 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:01.178715 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" event={"ID":"c228fa0a-24ca-410e-90ff-3fa2b82000e9","Type":"ContainerStarted","Data":"b09035b03df095691ab9d9fd9909649d0a3150ead413946132772b7208f583fd"} Nov 02 15:56:02.186098 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:02.186062 5476 generic.go:356] "Generic (PLEG): container finished" podID="c228fa0a-24ca-410e-90ff-3fa2b82000e9" containerID="e5874212bcb0d74c58bd92ecb47ef483f57b1165d3a5237755285cf83941a3b7" exitCode=0 Nov 02 15:56:02.186492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:02.186150 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" event={"ID":"c228fa0a-24ca-410e-90ff-3fa2b82000e9","Type":"ContainerDied","Data":"e5874212bcb0d74c58bd92ecb47ef483f57b1165d3a5237755285cf83941a3b7"} Nov 02 15:56:02.283920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:02.283890 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:56:03.360758 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:03.360721 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:03.451081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:03.450296 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-openshift-ca\") pod \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " Nov 02 15:56:03.451081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:03.450395 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-xks-ca\") pod \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " Nov 02 15:56:03.451081 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:03.450475 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c228fa0a-24ca-410e-90ff-3fa2b82000e9-kube-api-access-7t6ld\" (UniqueName: \"kubernetes.io/projected/c228fa0a-24ca-410e-90ff-3fa2b82000e9-kube-api-access-7t6ld\") pod \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\" (UID: \"c228fa0a-24ca-410e-90ff-3fa2b82000e9\") " Nov 02 15:56:03.455025 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:03.451522 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-openshift-ca" pod "c228fa0a-24ca-410e-90ff-3fa2b82000e9" (UID: "c228fa0a-24ca-410e-90ff-3fa2b82000e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:56:03.461159 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:03.461105 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-xks-ca" pod "c228fa0a-24ca-410e-90ff-3fa2b82000e9" (UID: "c228fa0a-24ca-410e-90ff-3fa2b82000e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:56:03.543019 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:03.517195 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c228fa0a-24ca-410e-90ff-3fa2b82000e9-kube-api-access-7t6ld" pod "c228fa0a-24ca-410e-90ff-3fa2b82000e9" (UID: "c228fa0a-24ca-410e-90ff-3fa2b82000e9"). InnerVolumeSpecName "kube-api-access-7t6ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:56:03.555735 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:03.555656 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t6ld\" (UniqueName: \"kubernetes.io/projected/c228fa0a-24ca-410e-90ff-3fa2b82000e9-kube-api-access-7t6ld\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:56:03.555735 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:03.555688 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:56:03.555735 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:03.555699 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c228fa0a-24ca-410e-90ff-3fa2b82000e9-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:56:04.228354 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:04.228258 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" event={"ID":"c228fa0a-24ca-410e-90ff-3fa2b82000e9","Type":"ContainerDied","Data":"b09035b03df095691ab9d9fd9909649d0a3150ead413946132772b7208f583fd"} Nov 02 15:56:04.228616 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:04.228593 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09035b03df095691ab9d9fd9909649d0a3150ead413946132772b7208f583fd" Nov 02 15:56:04.228803 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:04.228786 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452" Nov 02 15:56:12.984623 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:12.984592 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:56:16.868246 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:16.868210 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd"] Nov 02 15:56:16.913977 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:16.913948 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ftjfd"] Nov 02 15:56:16.984531 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:16.984496 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv"] Nov 02 15:56:16.984909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:16.984897 5476 cpu_manager.go:414] "containerMap: removing container" podUID="c228fa0a-24ca-410e-90ff-3fa2b82000e9" containerName="revoke-keys" Nov 02 15:56:16.984956 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:16.984910 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c228fa0a-24ca-410e-90ff-3fa2b82000e9" containerName="revoke-keys" Nov 02 15:56:16.985035 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:16.985014 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="c228fa0a-24ca-410e-90ff-3fa2b82000e9" containerName="revoke-keys" Nov 02 15:56:16.988281 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:16.988262 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:17.040804 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.040772 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:56:17.040804 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.040796 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:56:17.041030 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.040824 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:56:17.041030 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.040851 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:56:17.043919 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.043891 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv"] Nov 02 15:56:17.089739 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.089704 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1172d53-953c-4768-a092-e75331a4977d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8gqqv\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:17.089879 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.089814 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1172d53-953c-4768-a092-e75331a4977d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8gqqv\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:17.089879 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.089853 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d8xh\" (UniqueName: \"kubernetes.io/projected/d1172d53-953c-4768-a092-e75331a4977d-kube-api-access-6d8xh\") pod \"maas-api-revoke-sub-3d4156f1-8gqqv\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:17.190683 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.190595 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1172d53-953c-4768-a092-e75331a4977d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8gqqv\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:17.190864 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.190684 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1172d53-953c-4768-a092-e75331a4977d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8gqqv\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:17.190864 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.190718 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8xh\" (UniqueName: \"kubernetes.io/projected/d1172d53-953c-4768-a092-e75331a4977d-kube-api-access-6d8xh\") pod \"maas-api-revoke-sub-3d4156f1-8gqqv\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:17.191439 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.191421 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1172d53-953c-4768-a092-e75331a4977d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8gqqv\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:17.193199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.193177 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1172d53-953c-4768-a092-e75331a4977d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8gqqv\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:17.248249 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.248216 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d8xh\" (UniqueName: \"kubernetes.io/projected/d1172d53-953c-4768-a092-e75331a4977d-kube-api-access-6d8xh\") pod \"maas-api-revoke-sub-3d4156f1-8gqqv\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:17.300547 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.300521 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:17.674326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.674292 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv"] Nov 02 15:56:17.679631 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:56:17.679600 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1172d53_953c_4768_a092_e75331a4977d.slice/crio-ab85f4639922fe1d26b97c0668b27ca26d384bc60cd8bd47d66c93d75ab60dce WatchSource:0}: Error finding container ab85f4639922fe1d26b97c0668b27ca26d384bc60cd8bd47d66c93d75ab60dce: Status 404 returned error can't find the container with id ab85f4639922fe1d26b97c0668b27ca26d384bc60cd8bd47d66c93d75ab60dce Nov 02 15:56:17.921779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:17.921739 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3cc76da-29f3-4450-b4ba-676978e7d10b" path="/var/lib/kubelet/pods/c3cc76da-29f3-4450-b4ba-676978e7d10b/volumes" Nov 02 15:56:18.313373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:18.313277 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" event={"ID":"d1172d53-953c-4768-a092-e75331a4977d","Type":"ContainerStarted","Data":"f28d9f5fc4c8e5768076349f5acbc58e1a19bc388457d5a1055d7f54a5e3a0a8"} Nov 02 15:56:18.313373 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:18.313337 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" event={"ID":"d1172d53-953c-4768-a092-e75331a4977d","Type":"ContainerStarted","Data":"ab85f4639922fe1d26b97c0668b27ca26d384bc60cd8bd47d66c93d75ab60dce"} Nov 02 15:56:18.423800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:18.423743 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" podStartSLOduration=2.423728904 podStartE2EDuration="2.423728904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:56:18.422376533 +0000 UTC m=+3480.625254595" watchObservedRunningTime="2026-11-02 15:56:18.423728904 +0000 UTC m=+3480.626606961" Nov 02 15:56:19.046362 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:19.046323 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw"] Nov 02 15:56:19.091874 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:19.091840 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f4fnw"] Nov 02 15:56:19.321450 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:19.321369 5476 generic.go:356] "Generic (PLEG): container finished" podID="d1172d53-953c-4768-a092-e75331a4977d" containerID="f28d9f5fc4c8e5768076349f5acbc58e1a19bc388457d5a1055d7f54a5e3a0a8" exitCode=0 Nov 02 15:56:19.321598 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:19.321454 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" event={"ID":"d1172d53-953c-4768-a092-e75331a4977d","Type":"ContainerDied","Data":"f28d9f5fc4c8e5768076349f5acbc58e1a19bc388457d5a1055d7f54a5e3a0a8"} Nov 02 15:56:19.921104 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:19.921071 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa6a36d-087b-4135-8e9a-361365158a21" path="/var/lib/kubelet/pods/afa6a36d-087b-4135-8e9a-361365158a21/volumes" Nov 02 15:56:19.924429 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:19.924404 5476 scope.go:122] "RemoveContainer" containerID="d61f9917a614bada63af490f9f16f7988a29700baad7cf9dc3497d0a7204c688" Nov 02 15:56:20.146895 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:56:20.146859 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:56:20.461253 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:20.461222 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:20.623110 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:20.623078 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1172d53-953c-4768-a092-e75331a4977d-kube-api-access-6d8xh\" (UniqueName: \"kubernetes.io/projected/d1172d53-953c-4768-a092-e75331a4977d-kube-api-access-6d8xh\") pod \"d1172d53-953c-4768-a092-e75331a4977d\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " Nov 02 15:56:20.623296 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:20.623233 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1172d53-953c-4768-a092-e75331a4977d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1172d53-953c-4768-a092-e75331a4977d-maas-api-openshift-ca\") pod \"d1172d53-953c-4768-a092-e75331a4977d\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " Nov 02 15:56:20.623296 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:20.623257 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1172d53-953c-4768-a092-e75331a4977d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1172d53-953c-4768-a092-e75331a4977d-maas-api-xks-ca\") pod \"d1172d53-953c-4768-a092-e75331a4977d\" (UID: \"d1172d53-953c-4768-a092-e75331a4977d\") " Nov 02 15:56:20.623604 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:20.623579 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1172d53-953c-4768-a092-e75331a4977d-maas-api-openshift-ca" pod "d1172d53-953c-4768-a092-e75331a4977d" (UID: "d1172d53-953c-4768-a092-e75331a4977d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:56:20.624918 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:20.624898 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1172d53-953c-4768-a092-e75331a4977d-kube-api-access-6d8xh" pod "d1172d53-953c-4768-a092-e75331a4977d" (UID: "d1172d53-953c-4768-a092-e75331a4977d"). InnerVolumeSpecName "kube-api-access-6d8xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:56:20.625340 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:20.625325 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1172d53-953c-4768-a092-e75331a4977d-maas-api-xks-ca" pod "d1172d53-953c-4768-a092-e75331a4977d" (UID: "d1172d53-953c-4768-a092-e75331a4977d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:56:20.724408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:20.724370 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1172d53-953c-4768-a092-e75331a4977d-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:56:20.724408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:20.724402 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1172d53-953c-4768-a092-e75331a4977d-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:56:20.724408 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:20.724412 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d8xh\" (UniqueName: \"kubernetes.io/projected/d1172d53-953c-4768-a092-e75331a4977d-kube-api-access-6d8xh\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:56:21.334089 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:21.334052 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" event={"ID":"d1172d53-953c-4768-a092-e75331a4977d","Type":"ContainerDied","Data":"ab85f4639922fe1d26b97c0668b27ca26d384bc60cd8bd47d66c93d75ab60dce"} Nov 02 15:56:21.334496 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:21.334104 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab85f4639922fe1d26b97c0668b27ca26d384bc60cd8bd47d66c93d75ab60dce" Nov 02 15:56:21.334496 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:21.334060 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv" Nov 02 15:56:33.415512 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.415477 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx"] Nov 02 15:56:33.415894 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.415878 5476 cpu_manager.go:414] "containerMap: removing container" podUID="d1172d53-953c-4768-a092-e75331a4977d" containerName="revoke-keys" Nov 02 15:56:33.415894 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.415889 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1172d53-953c-4768-a092-e75331a4977d" containerName="revoke-keys" Nov 02 15:56:33.415986 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.415975 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1172d53-953c-4768-a092-e75331a4977d" containerName="revoke-keys" Nov 02 15:56:33.419446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.419420 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.446533 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.446502 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcpwj\" (UniqueName: \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-kube-api-access-bcpwj\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.446702 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.446567 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-credential-socket\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.446702 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.446626 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3654ab1-17bd-49e7-961c-9e513082bf13-istiod-ca-cert\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.446878 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.446776 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-certs\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.446968 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.446943 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-data\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.447055 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.447017 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-envoy\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.447187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.447163 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-podinfo\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.447394 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.447371 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-socket\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.447495 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.447408 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-token\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.491049 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.491020 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-a4b243-openshift-default-dockercfg-5k4q5\"" Nov 02 15:56:33.518824 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.518789 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx"] Nov 02 15:56:33.548410 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.548384 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpwj\" (UniqueName: \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-kube-api-access-bcpwj\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.548579 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.548437 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-credential-socket\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.548642 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.548568 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3654ab1-17bd-49e7-961c-9e513082bf13-istiod-ca-cert\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.548642 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.548621 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-certs\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.548761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.548676 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-data\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.548761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.548717 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-envoy\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.548761 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.548727 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-credential-socket\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.548950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.548775 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-podinfo\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.548950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.548805 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-socket\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.548950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.548832 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-token\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.549135 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.549007 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-certs\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.549135 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.549042 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-data\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.549262 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.549214 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-socket\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.549617 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.549596 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3654ab1-17bd-49e7-961c-9e513082bf13-istiod-ca-cert\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.551161 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.551134 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-envoy\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.551348 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.551329 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-podinfo\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.606209 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.606174 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-token\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.606388 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.606308 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpwj\" (UniqueName: \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-kube-api-access-bcpwj\") pod \"e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:33.732115 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:33.732035 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:34.161449 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:34.161414 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx"] Nov 02 15:56:34.166898 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:56:34.166868 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3654ab1_17bd_49e7_961c_9e513082bf13.slice/crio-dd0ca8bb965a9ebf8f350ee3768a9a8f696f650af1817157cffc97bd57a48ca3 WatchSource:0}: Error finding container dd0ca8bb965a9ebf8f350ee3768a9a8f696f650af1817157cffc97bd57a48ca3: Status 404 returned error can't find the container with id dd0ca8bb965a9ebf8f350ee3768a9a8f696f650af1817157cffc97bd57a48ca3 Nov 02 15:56:34.169188 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:34.169155 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:56:34.169270 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:34.169226 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:56:34.169270 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:34.169253 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:56:34.419485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:34.419391 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" event={"ID":"a3654ab1-17bd-49e7-961c-9e513082bf13","Type":"ContainerStarted","Data":"a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6"} Nov 02 15:56:34.419485 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:34.419460 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" event={"ID":"a3654ab1-17bd-49e7-961c-9e513082bf13","Type":"ContainerStarted","Data":"dd0ca8bb965a9ebf8f350ee3768a9a8f696f650af1817157cffc97bd57a48ca3"} Nov 02 15:56:34.536318 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:34.536271 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" podStartSLOduration=1.536258285 podStartE2EDuration="1.536258285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:56:34.533460642 +0000 UTC m=+3496.736338704" watchObservedRunningTime="2026-11-02 15:56:34.536258285 +0000 UTC m=+3496.739136344" Nov 02 15:56:34.732658 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:34.732585 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:35.737243 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:35.737216 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:36.430810 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:36.430769 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:36.431675 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:36.431656 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:56:43.371528 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.371487 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx"] Nov 02 15:56:43.375709 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.375685 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:43.445134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.445102 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4m69\" (UniqueName: \"kubernetes.io/projected/e3c51380-a052-4153-a507-863a23b5e381-kube-api-access-x4m69\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:43.445134 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.445137 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c51380-a052-4153-a507-863a23b5e381-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:43.445344 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.445161 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:43.445344 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.445302 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:43.470748 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.470721 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a4b243-serving-cert\"" Nov 02 15:56:43.470909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.470804 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 15:56:43.470909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.470809 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:56:43.470909 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.470844 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wj8b6\"" Nov 02 15:56:43.471138 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.471119 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:56:43.474249 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.474228 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx"] Nov 02 15:56:43.545980 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.545946 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:43.546187 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:56:43.546089 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-serving-cert: secret "maas-api-e2e-worker-single-main-a4b243-serving-cert" not found Nov 02 15:56:43.546187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.546108 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4m69\" (UniqueName: \"kubernetes.io/projected/e3c51380-a052-4153-a507-863a23b5e381-kube-api-access-x4m69\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:43.546187 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:56:43.546155 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-maas-api-tls podName:e3c51380-a052-4153-a507-863a23b5e381 nodeName:}" failed. No retries permitted until 2026-11-02 15:56:44.046131865 +0000 UTC m=+3506.249009903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-maas-api-tls") pod "maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" (UID: "e3c51380-a052-4153-a507-863a23b5e381") : secret "maas-api-e2e-worker-single-main-a4b243-serving-cert" not found Nov 02 15:56:43.546187 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.546178 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c51380-a052-4153-a507-863a23b5e381-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:43.546420 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.546224 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:43.546816 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.546797 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c51380-a052-4153-a507-863a23b5e381-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:43.548695 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.548674 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:43.603658 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:43.603626 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4m69\" (UniqueName: \"kubernetes.io/projected/e3c51380-a052-4153-a507-863a23b5e381-kube-api-access-x4m69\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:44.052288 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:44.052253 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:44.054672 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:44.054643 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:44.289302 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:44.289267 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:44.660011 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:44.659969 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx"] Nov 02 15:56:44.665923 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:56:44.665896 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c51380_a052_4153_a507_863a23b5e381.slice/crio-872f0a731f57200eb20f481d222a52100a166d24c8cf0342d55226cb13d1643d WatchSource:0}: Error finding container 872f0a731f57200eb20f481d222a52100a166d24c8cf0342d55226cb13d1643d: Status 404 returned error can't find the container with id 872f0a731f57200eb20f481d222a52100a166d24c8cf0342d55226cb13d1643d Nov 02 15:56:44.667139 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:44.667119 5476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:56:45.485211 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:45.485173 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" event={"ID":"e3c51380-a052-4153-a507-863a23b5e381","Type":"ContainerStarted","Data":"872f0a731f57200eb20f481d222a52100a166d24c8cf0342d55226cb13d1643d"} Nov 02 15:56:46.492661 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:46.492615 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" event={"ID":"e3c51380-a052-4153-a507-863a23b5e381","Type":"ContainerStarted","Data":"0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e"} Nov 02 15:56:46.493069 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:46.492693 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:46.606978 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:46.606935 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" podStartSLOduration=2.43724693 podStartE2EDuration="3.606923373s" totalImagesPullingTime="1.169676443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:56:46.606340808 +0000 UTC m=+3508.809218865" watchObservedRunningTime="2026-11-02 15:56:46.606923373 +0000 UTC m=+3508.809801433" Nov 02 15:56:47.047962 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:47.047927 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp"] Nov 02 15:56:47.052122 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:47.052098 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-zbphp"] Nov 02 15:56:47.921253 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:47.921219 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ed2a2f-7537-4808-a384-31487155491b" path="/var/lib/kubelet/pods/61ed2a2f-7537-4808-a384-31487155491b/volumes" Nov 02 15:56:50.147853 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:56:50.147809 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:56:51.294959 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:51.294918 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:56:52.505216 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:52.505183 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:56:59.054647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:59.054611 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q"] Nov 02 15:56:59.098966 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:59.098931 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-sg44q"] Nov 02 15:56:59.921827 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:56:59.921795 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb85ba13-2267-458c-9865-bf3a57841266" path="/var/lib/kubelet/pods/eb85ba13-2267-458c-9865-bf3a57841266/volumes" Nov 02 15:57:00.577647 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:00.577612 5476 generic.go:356] "Generic (PLEG): container finished" podID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerID="a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6" exitCode=0 Nov 02 15:57:00.578099 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:00.577700 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" event={"ID":"a3654ab1-17bd-49e7-961c-9e513082bf13","Type":"ContainerDied","Data":"a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6"} Nov 02 15:57:00.578175 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:00.578155 5476 scope.go:122] "RemoveContainer" containerID="a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6" Nov 02 15:57:00.579382 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:00.579355 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:57:00.579459 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:00.579425 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:57:00.579459 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:00.579455 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:57:01.603387 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:01.603343 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" event={"ID":"a3654ab1-17bd-49e7-961c-9e513082bf13","Type":"ContainerStarted","Data":"bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee"} Nov 02 15:57:01.732677 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:01.732640 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:57:01.732865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:01.732782 5476 patch_prober.go:29] interesting pod/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 02 15:57:01.732865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:01.732828 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 02 15:57:02.733356 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:02.733325 5476 patch_prober.go:29] interesting pod/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 02 15:57:02.733676 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:02.733392 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 02 15:57:03.733140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:03.733101 5476 patch_prober.go:29] interesting pod/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 02 15:57:03.733326 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:03.733169 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 02 15:57:04.732958 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:04.732923 5476 patch_prober.go:29] interesting pod/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 02 15:57:04.733397 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:04.732980 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 02 15:57:05.732734 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:05.732697 5476 patch_prober.go:29] interesting pod/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Nov 02 15:57:05.732918 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:05.732747 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Nov 02 15:57:06.430950 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:06.430916 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:57:06.736225 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:06.736147 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:57:06.737107 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:06.737085 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:57:10.278848 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:10.278804 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:57:19.948332 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:19.948301 5476 scope.go:122] "RemoveContainer" containerID="97e1f761708c0d889a5eb8ce73f377b2f6d2e9cf01ba5a4601ea2aa32b2101dd" Nov 02 15:57:20.146385 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:57:20.146350 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:57:35.057500 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:35.057468 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2"] Nov 02 15:57:35.103547 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:35.103519 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-qjxq2"] Nov 02 15:57:35.787738 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:35.787700 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:57:35.923189 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:35.923150 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3dea9e7-c2ad-4954-a285-e6beba1f4e01" path="/var/lib/kubelet/pods/b3dea9e7-c2ad-4954-a285-e6beba1f4e01/volumes" Nov 02 15:57:47.048537 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:47.048504 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b"] Nov 02 15:57:47.094729 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:47.094697 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-dv58b"] Nov 02 15:57:47.921848 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:57:47.921813 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1c909e-78a8-48f4-a28b-fbb1870c059b" path="/var/lib/kubelet/pods/ef1c909e-78a8-48f4-a28b-fbb1870c059b/volumes" Nov 02 15:57:50.145835 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:57:50.145799 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:58:04.982732 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:04.982699 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:58:18.302017 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:18.301963 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx"] Nov 02 15:58:18.302481 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:18.302151 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" podUID="e3c51380-a052-4153-a507-863a23b5e381" containerName="maas-api" containerID="cri-o://0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e" gracePeriod=30 Nov 02 15:58:18.747811 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:18.747770 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:58:18.755814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:18.755789 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 15:58:19.554295 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.554272 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:58:19.654331 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.654254 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-maas-api-tls\") pod \"e3c51380-a052-4153-a507-863a23b5e381\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " Nov 02 15:58:19.654492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.654335 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c51380-a052-4153-a507-863a23b5e381-kube-api-access-x4m69\" (UniqueName: \"kubernetes.io/projected/e3c51380-a052-4153-a507-863a23b5e381-kube-api-access-x4m69\") pod \"e3c51380-a052-4153-a507-863a23b5e381\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " Nov 02 15:58:19.654492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.654355 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c51380-a052-4153-a507-863a23b5e381-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c51380-a052-4153-a507-863a23b5e381-openshift-service-ca\") pod \"e3c51380-a052-4153-a507-863a23b5e381\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " Nov 02 15:58:19.654492 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.654388 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-metrics-certs\") pod \"e3c51380-a052-4153-a507-863a23b5e381\" (UID: \"e3c51380-a052-4153-a507-863a23b5e381\") " Nov 02 15:58:19.654754 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.654729 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c51380-a052-4153-a507-863a23b5e381-openshift-service-ca" pod "e3c51380-a052-4153-a507-863a23b5e381" (UID: "e3c51380-a052-4153-a507-863a23b5e381"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:58:19.656529 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.656502 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c51380-a052-4153-a507-863a23b5e381-kube-api-access-x4m69" pod "e3c51380-a052-4153-a507-863a23b5e381" (UID: "e3c51380-a052-4153-a507-863a23b5e381"). InnerVolumeSpecName "kube-api-access-x4m69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:58:19.656529 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.656514 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-metrics-certs" pod "e3c51380-a052-4153-a507-863a23b5e381" (UID: "e3c51380-a052-4153-a507-863a23b5e381"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:58:19.656687 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.656537 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-maas-api-tls" pod "e3c51380-a052-4153-a507-863a23b5e381" (UID: "e3c51380-a052-4153-a507-863a23b5e381"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:58:19.755286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.755250 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:19.755286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.755283 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4m69\" (UniqueName: \"kubernetes.io/projected/e3c51380-a052-4153-a507-863a23b5e381-kube-api-access-x4m69\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:19.755286 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.755294 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c51380-a052-4153-a507-863a23b5e381-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:19.755545 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.755302 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3c51380-a052-4153-a507-863a23b5e381-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:19.971518 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.971484 5476 scope.go:122] "RemoveContainer" containerID="d947cf1d8f1457b58e553322a7965667f54bdd88da6bab880fce13608756fcbc" Nov 02 15:58:19.981199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:19.981179 5476 scope.go:122] "RemoveContainer" containerID="309a5b81c5882498d432468963fd7945c52694ed9258d71b829a3edb575097f8" Nov 02 15:58:20.067800 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:20.067759 5476 generic.go:356] "Generic (PLEG): container finished" podID="e3c51380-a052-4153-a507-863a23b5e381" containerID="0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e" exitCode=0 Nov 02 15:58:20.068032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:20.067812 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" Nov 02 15:58:20.068032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:20.067841 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" event={"ID":"e3c51380-a052-4153-a507-863a23b5e381","Type":"ContainerDied","Data":"0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e"} Nov 02 15:58:20.068032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:20.067896 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx" event={"ID":"e3c51380-a052-4153-a507-863a23b5e381","Type":"ContainerDied","Data":"872f0a731f57200eb20f481d222a52100a166d24c8cf0342d55226cb13d1643d"} Nov 02 15:58:20.068032 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:20.067922 5476 scope.go:122] "RemoveContainer" containerID="0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e" Nov 02 15:58:20.077838 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:20.077801 5476 scope.go:122] "RemoveContainer" containerID="0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e" Nov 02 15:58:20.078181 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:58:20.078156 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e\": container with ID starting with 0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e not found: ID does not exist" containerID="0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e" Nov 02 15:58:20.078284 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:20.078193 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e"} err="failed to get container status \"0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e\": rpc error: code = NotFound desc = could not find container \"0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e\": container with ID starting with 0baa1849a67a6bb325ce092fe592066859fb34bc419eddd753770401ce33282e not found: ID does not exist" Nov 02 15:58:20.146300 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:58:20.146262 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:58:20.225659 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:20.225581 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx"] Nov 02 15:58:20.271629 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:20.271596 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a4b243-86867bfdf4-lzvcx"] Nov 02 15:58:21.046056 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:21.046016 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76"] Nov 02 15:58:21.092262 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:21.092229 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-6qg76"] Nov 02 15:58:21.921469 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:21.921437 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1e11201-ff4d-405e-bd68-b4141e2bab8b" path="/var/lib/kubelet/pods/b1e11201-ff4d-405e-bd68-b4141e2bab8b/volumes" Nov 02 15:58:21.921850 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:21.921836 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c51380-a052-4153-a507-863a23b5e381" path="/var/lib/kubelet/pods/e3c51380-a052-4153-a507-863a23b5e381/volumes" Nov 02 15:58:22.988248 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:22.988213 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 15:58:27.432199 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:27.431986 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx"] Nov 02 15:58:27.432982 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:27.432922 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" containerID="cri-o://bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee" gracePeriod=30 Nov 02 15:58:28.948858 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:28.948825 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv"] Nov 02 15:58:28.994920 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:28.994891 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8gqqv"] Nov 02 15:58:29.066215 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.066185 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8"] Nov 02 15:58:29.066630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.066609 5476 cpu_manager.go:414] "containerMap: removing container" podUID="e3c51380-a052-4153-a507-863a23b5e381" containerName="maas-api" Nov 02 15:58:29.066630 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.066624 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c51380-a052-4153-a507-863a23b5e381" containerName="maas-api" Nov 02 15:58:29.066726 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.066716 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c51380-a052-4153-a507-863a23b5e381" containerName="maas-api" Nov 02 15:58:29.070428 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.070410 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:29.119517 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.119487 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:58:29.119664 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.119522 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:58:29.119664 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.119581 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:58:29.119664 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.119584 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:58:29.127905 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.127841 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8"] Nov 02 15:58:29.137962 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.137933 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-56lw8\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:29.138095 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.137978 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-56lw8\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:29.138140 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.138105 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ftjz\" (UniqueName: \"kubernetes.io/projected/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-kube-api-access-7ftjz\") pod \"maas-api-revoke-sub-3d4156f1-56lw8\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:29.239610 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.239534 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-56lw8\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:29.239610 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.239569 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-56lw8\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:29.239610 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.239602 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ftjz\" (UniqueName: \"kubernetes.io/projected/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-kube-api-access-7ftjz\") pod \"maas-api-revoke-sub-3d4156f1-56lw8\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:29.240337 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.240311 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-56lw8\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:29.241969 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.241944 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-56lw8\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:29.297475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.297448 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ftjz\" (UniqueName: \"kubernetes.io/projected/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-kube-api-access-7ftjz\") pod \"maas-api-revoke-sub-3d4156f1-56lw8\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:29.382894 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.382862 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:29.752480 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.752447 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8"] Nov 02 15:58:29.758341 ip-10-0-142-141 kubenswrapper[5476]: W1102 15:58:29.758307 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5a5ee7_a423_40fa_b0d5_5e8817b753c3.slice/crio-b3c32e398c6d0890670d0c0ac3c24dabedf0e40f7a5827b30fb9c9b519e4f40c WatchSource:0}: Error finding container b3c32e398c6d0890670d0c0ac3c24dabedf0e40f7a5827b30fb9c9b519e4f40c: Status 404 returned error can't find the container with id b3c32e398c6d0890670d0c0ac3c24dabedf0e40f7a5827b30fb9c9b519e4f40c Nov 02 15:58:29.925183 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:29.922788 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1172d53-953c-4768-a092-e75331a4977d" path="/var/lib/kubelet/pods/d1172d53-953c-4768-a092-e75331a4977d/volumes" Nov 02 15:58:30.138429 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:30.138391 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" event={"ID":"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3","Type":"ContainerStarted","Data":"3a29dde030b9e750f7f68d62b2d439967594720ce3efdf3f9d538a99f8b86c18"} Nov 02 15:58:30.138779 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:30.138450 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" event={"ID":"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3","Type":"ContainerStarted","Data":"b3c32e398c6d0890670d0c0ac3c24dabedf0e40f7a5827b30fb9c9b519e4f40c"} Nov 02 15:58:30.248866 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:30.248805 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" podStartSLOduration=1.248787337 podStartE2EDuration="1.248787337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:58:30.246570329 +0000 UTC m=+3612.449448388" watchObservedRunningTime="2026-11-02 15:58:30.248787337 +0000 UTC m=+3612.451665398" Nov 02 15:58:31.145562 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:31.145531 5476 generic.go:356] "Generic (PLEG): container finished" podID="ad5a5ee7-a423-40fa-b0d5-5e8817b753c3" containerID="3a29dde030b9e750f7f68d62b2d439967594720ce3efdf3f9d538a99f8b86c18" exitCode=0 Nov 02 15:58:31.145959 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:31.145587 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" event={"ID":"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3","Type":"ContainerDied","Data":"3a29dde030b9e750f7f68d62b2d439967594720ce3efdf3f9d538a99f8b86c18"} Nov 02 15:58:32.294440 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.294413 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:32.371310 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.371283 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-openshift-ca\") pod \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " Nov 02 15:58:32.371446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.371315 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-xks-ca\") pod \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " Nov 02 15:58:32.371446 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.371375 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-kube-api-access-7ftjz\" (UniqueName: \"kubernetes.io/projected/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-kube-api-access-7ftjz\") pod \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\" (UID: \"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3\") " Nov 02 15:58:32.371635 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.371614 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-openshift-ca" pod "ad5a5ee7-a423-40fa-b0d5-5e8817b753c3" (UID: "ad5a5ee7-a423-40fa-b0d5-5e8817b753c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:58:32.373394 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.373368 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-xks-ca" pod "ad5a5ee7-a423-40fa-b0d5-5e8817b753c3" (UID: "ad5a5ee7-a423-40fa-b0d5-5e8817b753c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:58:32.373510 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.373478 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-kube-api-access-7ftjz" pod "ad5a5ee7-a423-40fa-b0d5-5e8817b753c3" (UID: "ad5a5ee7-a423-40fa-b0d5-5e8817b753c3"). InnerVolumeSpecName "kube-api-access-7ftjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:58:32.472619 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.472595 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:32.472724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.472621 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:32.472724 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.472632 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ftjz\" (UniqueName: \"kubernetes.io/projected/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3-kube-api-access-7ftjz\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:32.698911 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.698854 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:58:32.775434 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775407 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3654ab1-17bd-49e7-961c-9e513082bf13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3654ab1-17bd-49e7-961c-9e513082bf13-istiod-ca-cert\") pod \"a3654ab1-17bd-49e7-961c-9e513082bf13\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " Nov 02 15:58:32.775583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775461 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-data\") pod \"a3654ab1-17bd-49e7-961c-9e513082bf13\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " Nov 02 15:58:32.775583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775485 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-credential-socket\") pod \"a3654ab1-17bd-49e7-961c-9e513082bf13\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " Nov 02 15:58:32.775583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775539 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-socket\") pod \"a3654ab1-17bd-49e7-961c-9e513082bf13\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " Nov 02 15:58:32.775583 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775568 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-envoy\") pod \"a3654ab1-17bd-49e7-961c-9e513082bf13\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " Nov 02 15:58:32.775814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775607 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-kube-api-access-bcpwj\" (UniqueName: \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-kube-api-access-bcpwj\") pod \"a3654ab1-17bd-49e7-961c-9e513082bf13\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " Nov 02 15:58:32.775814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775654 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-certs\") pod \"a3654ab1-17bd-49e7-961c-9e513082bf13\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " Nov 02 15:58:32.775814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775710 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-podinfo\") pod \"a3654ab1-17bd-49e7-961c-9e513082bf13\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " Nov 02 15:58:32.775814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775728 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-data" pod "a3654ab1-17bd-49e7-961c-9e513082bf13" (UID: "a3654ab1-17bd-49e7-961c-9e513082bf13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:58:32.775814 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775736 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-token\" (UniqueName: \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-token\") pod \"a3654ab1-17bd-49e7-961c-9e513082bf13\" (UID: \"a3654ab1-17bd-49e7-961c-9e513082bf13\") " Nov 02 15:58:32.776118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775873 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3654ab1-17bd-49e7-961c-9e513082bf13-istiod-ca-cert" pod "a3654ab1-17bd-49e7-961c-9e513082bf13" (UID: "a3654ab1-17bd-49e7-961c-9e513082bf13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:58:32.776118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775907 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-credential-socket" pod "a3654ab1-17bd-49e7-961c-9e513082bf13" (UID: "a3654ab1-17bd-49e7-961c-9e513082bf13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:58:32.776118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.775906 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-socket" pod "a3654ab1-17bd-49e7-961c-9e513082bf13" (UID: "a3654ab1-17bd-49e7-961c-9e513082bf13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:58:32.776118 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.776085 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-certs" pod "a3654ab1-17bd-49e7-961c-9e513082bf13" (UID: "a3654ab1-17bd-49e7-961c-9e513082bf13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:58:32.776328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.776218 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:32.776328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.776241 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3654ab1-17bd-49e7-961c-9e513082bf13-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:32.776328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.776254 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:32.776328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.776268 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:32.776328 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.776280 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:32.777875 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.777847 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-podinfo" pod "a3654ab1-17bd-49e7-961c-9e513082bf13" (UID: "a3654ab1-17bd-49e7-961c-9e513082bf13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:58:32.778015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.777912 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-envoy" pod "a3654ab1-17bd-49e7-961c-9e513082bf13" (UID: "a3654ab1-17bd-49e7-961c-9e513082bf13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:58:32.778015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.777935 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-kube-api-access-bcpwj" pod "a3654ab1-17bd-49e7-961c-9e513082bf13" (UID: "a3654ab1-17bd-49e7-961c-9e513082bf13"). InnerVolumeSpecName "kube-api-access-bcpwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:58:32.778015 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.777959 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-token" pod "a3654ab1-17bd-49e7-961c-9e513082bf13" (UID: "a3654ab1-17bd-49e7-961c-9e513082bf13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:58:32.877351 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.877313 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:32.877351 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.877347 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcpwj\" (UniqueName: \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-kube-api-access-bcpwj\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:32.877539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.877358 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:32.877539 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:32.877367 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3654ab1-17bd-49e7-961c-9e513082bf13-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:33.157595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.157565 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" Nov 02 15:58:33.157595 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.157575 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8" event={"ID":"ad5a5ee7-a423-40fa-b0d5-5e8817b753c3","Type":"ContainerDied","Data":"b3c32e398c6d0890670d0c0ac3c24dabedf0e40f7a5827b30fb9c9b519e4f40c"} Nov 02 15:58:33.157865 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.157624 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3c32e398c6d0890670d0c0ac3c24dabedf0e40f7a5827b30fb9c9b519e4f40c" Nov 02 15:58:33.159488 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.159457 5476 generic.go:356] "Generic (PLEG): container finished" podID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerID="bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee" exitCode=0 Nov 02 15:58:33.159634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.159508 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" Nov 02 15:58:33.159634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.159542 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" event={"ID":"a3654ab1-17bd-49e7-961c-9e513082bf13","Type":"ContainerDied","Data":"bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee"} Nov 02 15:58:33.159634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.159603 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx" event={"ID":"a3654ab1-17bd-49e7-961c-9e513082bf13","Type":"ContainerDied","Data":"dd0ca8bb965a9ebf8f350ee3768a9a8f696f650af1817157cffc97bd57a48ca3"} Nov 02 15:58:33.159634 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.159632 5476 scope.go:122] "RemoveContainer" containerID="bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee" Nov 02 15:58:33.170465 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.170447 5476 scope.go:122] "RemoveContainer" containerID="a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6" Nov 02 15:58:33.179399 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.179377 5476 scope.go:122] "RemoveContainer" containerID="bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee" Nov 02 15:58:33.179665 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:58:33.179646 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee\": container with ID starting with bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee not found: ID does not exist" containerID="bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee" Nov 02 15:58:33.179713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.179674 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee"} err="failed to get container status \"bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee\": rpc error: code = NotFound desc = could not find container \"bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee\": container with ID starting with bfb58e3e82202b615d95fa960e0c5fc2b7c18c8f3961f61abd2d1aaabc5747ee not found: ID does not exist" Nov 02 15:58:33.179713 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.179690 5476 scope.go:122] "RemoveContainer" containerID="a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6" Nov 02 15:58:33.179953 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:58:33.179930 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6\": container with ID starting with a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6 not found: ID does not exist" containerID="a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6" Nov 02 15:58:33.180013 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.179960 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6"} err="failed to get container status \"a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6\": rpc error: code = NotFound desc = could not find container \"a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6\": container with ID starting with a2920352f94800a90ef2881c235e665e05b4bc0bcda3bfdc4cfacdc901733bc6 not found: ID does not exist" Nov 02 15:58:33.316475 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.316454 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx"] Nov 02 15:58:33.362823 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.362797 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-a4b243-openshift-default-79d89f4cf7w6bgx"] Nov 02 15:58:33.921133 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:33.921102 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" path="/var/lib/kubelet/pods/a3654ab1-17bd-49e7-961c-9e513082bf13/volumes" Nov 02 15:58:48.048380 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:48.048344 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm"] Nov 02 15:58:48.094363 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:48.094332 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-pcdwm"] Nov 02 15:58:49.922901 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:58:49.922863 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71dc5cad-967b-48f2-8488-655503afa259" path="/var/lib/kubelet/pods/71dc5cad-967b-48f2-8488-655503afa259/volumes" Nov 02 15:58:50.146452 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:58:50.146416 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:59:19.046777 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:59:19.046742 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s"] Nov 02 15:59:19.093020 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:59:19.092975 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-94m2s"] Nov 02 15:59:19.920393 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:59:19.920360 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42d847d-7962-4dff-b754-f4e1368d8070" path="/var/lib/kubelet/pods/d42d847d-7962-4dff-b754-f4e1368d8070/volumes" Nov 02 15:59:20.008327 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:59:20.008297 5476 scope.go:122] "RemoveContainer" containerID="8b8b3c42f4b24a3aaeb598d2af85b9032f1836cabca5bc9b85d83ca6ac8432ae" Nov 02 15:59:20.017681 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:59:20.017651 5476 scope.go:122] "RemoveContainer" containerID="eb11f0d04ffb2b769d561fb13819e88c3c63991356496f6a75373e1f793d6f5f" Nov 02 15:59:20.028467 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:59:20.028448 5476 scope.go:122] "RemoveContainer" containerID="4306674b28bd74adaa96646cd955268dba483ed61b1b3216cab4f8c047e5921e" Nov 02 15:59:20.145532 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:59:20.145500 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 15:59:42.116034 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:59:42.115986 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp"] Nov 02 15:59:42.160968 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:59:42.160938 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-24nfp"] Nov 02 15:59:43.921695 ip-10-0-142-141 kubenswrapper[5476]: I1102 15:59:43.921654 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2507e56-412b-429e-8703-bc465bdcf325" path="/var/lib/kubelet/pods/b2507e56-412b-429e-8703-bc465bdcf325/volumes" Nov 02 15:59:50.146706 ip-10-0-142-141 kubenswrapper[5476]: E1102 15:59:50.146662 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:00:00.167105 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.167019 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g"] Nov 02 16:00:00.167529 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.167405 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ad5a5ee7-a423-40fa-b0d5-5e8817b753c3" containerName="revoke-keys" Nov 02 16:00:00.167529 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.167416 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5a5ee7-a423-40fa-b0d5-5e8817b753c3" containerName="revoke-keys" Nov 02 16:00:00.167529 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.167429 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" Nov 02 16:00:00.167529 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.167434 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" Nov 02 16:00:00.167529 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.167470 5476 cpu_manager.go:414] "containerMap: removing container" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" Nov 02 16:00:00.167529 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.167475 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" Nov 02 16:00:00.167752 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.167536 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" Nov 02 16:00:00.167752 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.167548 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5a5ee7-a423-40fa-b0d5-5e8817b753c3" containerName="revoke-keys" Nov 02 16:00:00.170779 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.170762 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" Nov 02 16:00:00.220083 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.220048 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:00:00.266282 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.266248 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:00:00.266282 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.266258 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 16:00:00.268353 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.268331 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g"] Nov 02 16:00:00.305107 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.305074 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmpj\" (UniqueName: \"kubernetes.io/projected/fc1c626d-1d1d-4a54-a175-83c56371e8cf-kube-api-access-dbmpj\") pod \"maas-api-key-cleanup-29893920-th56g\" (UID: \"fc1c626d-1d1d-4a54-a175-83c56371e8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" Nov 02 16:00:00.305268 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.305198 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1c626d-1d1d-4a54-a175-83c56371e8cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893920-th56g\" (UID: \"fc1c626d-1d1d-4a54-a175-83c56371e8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" Nov 02 16:00:00.406414 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.406374 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmpj\" (UniqueName: \"kubernetes.io/projected/fc1c626d-1d1d-4a54-a175-83c56371e8cf-kube-api-access-dbmpj\") pod \"maas-api-key-cleanup-29893920-th56g\" (UID: \"fc1c626d-1d1d-4a54-a175-83c56371e8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" Nov 02 16:00:00.406604 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.406476 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1c626d-1d1d-4a54-a175-83c56371e8cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893920-th56g\" (UID: \"fc1c626d-1d1d-4a54-a175-83c56371e8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" Nov 02 16:00:00.407127 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.407106 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1c626d-1d1d-4a54-a175-83c56371e8cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893920-th56g\" (UID: \"fc1c626d-1d1d-4a54-a175-83c56371e8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" Nov 02 16:00:00.464223 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.464145 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmpj\" (UniqueName: \"kubernetes.io/projected/fc1c626d-1d1d-4a54-a175-83c56371e8cf-kube-api-access-dbmpj\") pod \"maas-api-key-cleanup-29893920-th56g\" (UID: \"fc1c626d-1d1d-4a54-a175-83c56371e8cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" Nov 02 16:00:00.483096 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.483060 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" Nov 02 16:00:00.851282 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:00.851206 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g"] Nov 02 16:00:00.857234 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:00:00.857206 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1c626d_1d1d_4a54_a175_83c56371e8cf.slice/crio-727fa9f66e25a8981a9a6909c63ce47a2e7c0b1f7efaf628e7be4dbb876382fc WatchSource:0}: Error finding container 727fa9f66e25a8981a9a6909c63ce47a2e7c0b1f7efaf628e7be4dbb876382fc: Status 404 returned error can't find the container with id 727fa9f66e25a8981a9a6909c63ce47a2e7c0b1f7efaf628e7be4dbb876382fc Nov 02 16:00:01.659232 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:01.659178 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" event={"ID":"fc1c626d-1d1d-4a54-a175-83c56371e8cf","Type":"ContainerStarted","Data":"476f7c4d0d2e1f5ae6d54f169ea0ae9bd207a1290791227f5548d1d4e0c5948a"} Nov 02 16:00:01.659633 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:01.659290 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" event={"ID":"fc1c626d-1d1d-4a54-a175-83c56371e8cf","Type":"ContainerStarted","Data":"727fa9f66e25a8981a9a6909c63ce47a2e7c0b1f7efaf628e7be4dbb876382fc"} Nov 02 16:00:01.769120 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:01.769061 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" podStartSLOduration=1.769049007 podStartE2EDuration="1.769049007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:00:01.766850843 +0000 UTC m=+3703.969728899" watchObservedRunningTime="2026-11-02 16:00:01.769049007 +0000 UTC m=+3703.971927070" Nov 02 16:00:02.666319 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:02.666285 5476 generic.go:356] "Generic (PLEG): container finished" podID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerID="476f7c4d0d2e1f5ae6d54f169ea0ae9bd207a1290791227f5548d1d4e0c5948a" exitCode=60 Nov 02 16:00:02.666785 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:02.666353 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" event={"ID":"fc1c626d-1d1d-4a54-a175-83c56371e8cf","Type":"ContainerDied","Data":"476f7c4d0d2e1f5ae6d54f169ea0ae9bd207a1290791227f5548d1d4e0c5948a"} Nov 02 16:00:02.666785 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:02.666575 5476 scope.go:122] "RemoveContainer" containerID="476f7c4d0d2e1f5ae6d54f169ea0ae9bd207a1290791227f5548d1d4e0c5948a" Nov 02 16:00:03.674132 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:03.674100 5476 generic.go:356] "Generic (PLEG): container finished" podID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerID="cf7533c0ff7ada377d7affeb9278d32d858ecd507f5ea7728e890b33dc772ed3" exitCode=60 Nov 02 16:00:03.674598 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:03.674183 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" event={"ID":"fc1c626d-1d1d-4a54-a175-83c56371e8cf","Type":"ContainerDied","Data":"cf7533c0ff7ada377d7affeb9278d32d858ecd507f5ea7728e890b33dc772ed3"} Nov 02 16:00:03.674598 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:03.674256 5476 scope.go:122] "RemoveContainer" containerID="476f7c4d0d2e1f5ae6d54f169ea0ae9bd207a1290791227f5548d1d4e0c5948a" Nov 02 16:00:03.674598 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:03.674461 5476 scope.go:122] "RemoveContainer" containerID="cf7533c0ff7ada377d7affeb9278d32d858ecd507f5ea7728e890b33dc772ed3" Nov 02 16:00:03.674716 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:00:03.674692 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893920-th56g_odh-ai-gateway-infra(fc1c626d-1d1d-4a54-a175-83c56371e8cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" Nov 02 16:00:04.682538 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:04.682505 5476 scope.go:122] "RemoveContainer" containerID="cf7533c0ff7ada377d7affeb9278d32d858ecd507f5ea7728e890b33dc772ed3" Nov 02 16:00:04.683024 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:00:04.682788 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893920-th56g_odh-ai-gateway-infra(fc1c626d-1d1d-4a54-a175-83c56371e8cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" Nov 02 16:00:12.882224 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:12.882188 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6"] Nov 02 16:00:12.882696 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:12.882667 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3654ab1-17bd-49e7-961c-9e513082bf13" containerName="istio-proxy" Nov 02 16:00:12.886907 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:12.886884 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" Nov 02 16:00:12.938382 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:12.938344 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 16:00:12.986803 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:12.986768 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6"] Nov 02 16:00:12.999112 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:12.999070 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 16:00:13.037352 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.037316 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-k4hl6\" (UID: \"a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" Nov 02 16:00:13.037352 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.037359 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-k4hl6\" (UID: \"a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" Nov 02 16:00:13.037586 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.037392 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlx6n\" (UniqueName: \"kubernetes.io/projected/a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29-kube-api-access-wlx6n\") pod \"kuadrant-console-plugin-66c4fb575d-k4hl6\" (UID: \"a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" Nov 02 16:00:13.138565 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.138477 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-k4hl6\" (UID: \"a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" Nov 02 16:00:13.138565 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.138529 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-k4hl6\" (UID: \"a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" Nov 02 16:00:13.138767 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.138574 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlx6n\" (UniqueName: \"kubernetes.io/projected/a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29-kube-api-access-wlx6n\") pod \"kuadrant-console-plugin-66c4fb575d-k4hl6\" (UID: \"a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" Nov 02 16:00:13.139362 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.139343 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-k4hl6\" (UID: \"a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" Nov 02 16:00:13.141115 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.141091 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-k4hl6\" (UID: \"a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" Nov 02 16:00:13.173037 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.172987 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:00:13.196244 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.196213 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlx6n\" (UniqueName: \"kubernetes.io/projected/a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29-kube-api-access-wlx6n\") pod \"kuadrant-console-plugin-66c4fb575d-k4hl6\" (UID: \"a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" Nov 02 16:00:13.199087 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.199063 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" Nov 02 16:00:13.569248 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.569151 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6"] Nov 02 16:00:13.574842 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:00:13.574801 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7a05c4e_b333_4e50_b2ad_0cdfa09bdf29.slice/crio-5d2af205a0b1ea78fd8f99da2bd5cddc80349264b4e3eaa1dfad7649c6c9ddef WatchSource:0}: Error finding container 5d2af205a0b1ea78fd8f99da2bd5cddc80349264b4e3eaa1dfad7649c6c9ddef: Status 404 returned error can't find the container with id 5d2af205a0b1ea78fd8f99da2bd5cddc80349264b4e3eaa1dfad7649c6c9ddef Nov 02 16:00:13.735913 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.735871 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" event={"ID":"a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29","Type":"ContainerStarted","Data":"5d2af205a0b1ea78fd8f99da2bd5cddc80349264b4e3eaa1dfad7649c6c9ddef"} Nov 02 16:00:13.915747 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:13.915707 5476 scope.go:122] "RemoveContainer" containerID="cf7533c0ff7ada377d7affeb9278d32d858ecd507f5ea7728e890b33dc772ed3" Nov 02 16:00:14.751107 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:14.751071 5476 generic.go:356] "Generic (PLEG): container finished" podID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerID="e0c637acb622f0ebc16d0e4fe0e7dbf09cf51813dfa85e34c46acf7bace75a8c" exitCode=60 Nov 02 16:00:14.751417 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:14.751245 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" event={"ID":"fc1c626d-1d1d-4a54-a175-83c56371e8cf","Type":"ContainerDied","Data":"e0c637acb622f0ebc16d0e4fe0e7dbf09cf51813dfa85e34c46acf7bace75a8c"} Nov 02 16:00:14.751564 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:14.751553 5476 scope.go:122] "RemoveContainer" containerID="cf7533c0ff7ada377d7affeb9278d32d858ecd507f5ea7728e890b33dc772ed3" Nov 02 16:00:14.752492 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:14.752476 5476 scope.go:122] "RemoveContainer" containerID="e0c637acb622f0ebc16d0e4fe0e7dbf09cf51813dfa85e34c46acf7bace75a8c" Nov 02 16:00:14.753681 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:00:14.753214 5476 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893920-th56g_odh-ai-gateway-infra(fc1c626d-1d1d-4a54-a175-83c56371e8cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" Nov 02 16:00:15.889021 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:15.883734 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g"] Nov 02 16:00:16.151734 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.151653 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" Nov 02 16:00:16.170838 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.170804 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1c626d-1d1d-4a54-a175-83c56371e8cf-kube-api-access-dbmpj\" (UniqueName: \"kubernetes.io/projected/fc1c626d-1d1d-4a54-a175-83c56371e8cf-kube-api-access-dbmpj\") pod \"fc1c626d-1d1d-4a54-a175-83c56371e8cf\" (UID: \"fc1c626d-1d1d-4a54-a175-83c56371e8cf\") " Nov 02 16:00:16.171069 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.170908 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1c626d-1d1d-4a54-a175-83c56371e8cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1c626d-1d1d-4a54-a175-83c56371e8cf-maas-api-service-ca\") pod \"fc1c626d-1d1d-4a54-a175-83c56371e8cf\" (UID: \"fc1c626d-1d1d-4a54-a175-83c56371e8cf\") " Nov 02 16:00:16.171323 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.171295 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1c626d-1d1d-4a54-a175-83c56371e8cf-maas-api-service-ca" pod "fc1c626d-1d1d-4a54-a175-83c56371e8cf" (UID: "fc1c626d-1d1d-4a54-a175-83c56371e8cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:00:16.173110 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.173082 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1c626d-1d1d-4a54-a175-83c56371e8cf-kube-api-access-dbmpj" pod "fc1c626d-1d1d-4a54-a175-83c56371e8cf" (UID: "fc1c626d-1d1d-4a54-a175-83c56371e8cf"). InnerVolumeSpecName "kube-api-access-dbmpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:00:16.272090 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.272041 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbmpj\" (UniqueName: \"kubernetes.io/projected/fc1c626d-1d1d-4a54-a175-83c56371e8cf-kube-api-access-dbmpj\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:00:16.272090 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.272081 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc1c626d-1d1d-4a54-a175-83c56371e8cf-maas-api-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:00:16.765852 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.765812 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" event={"ID":"fc1c626d-1d1d-4a54-a175-83c56371e8cf","Type":"ContainerDied","Data":"727fa9f66e25a8981a9a6909c63ce47a2e7c0b1f7efaf628e7be4dbb876382fc"} Nov 02 16:00:16.765852 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.765848 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g" Nov 02 16:00:16.766157 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.765913 5476 scope.go:122] "RemoveContainer" containerID="e0c637acb622f0ebc16d0e4fe0e7dbf09cf51813dfa85e34c46acf7bace75a8c" Nov 02 16:00:16.919673 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.919629 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g"] Nov 02 16:00:16.923824 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:16.923797 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-th56g"] Nov 02 16:00:17.451680 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.451645 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4"] Nov 02 16:00:17.452085 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.452070 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerName="cleanup" Nov 02 16:00:17.452147 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.452089 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerName="cleanup" Nov 02 16:00:17.452147 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.452107 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerName="cleanup" Nov 02 16:00:17.452147 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.452112 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerName="cleanup" Nov 02 16:00:17.452147 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.452135 5476 cpu_manager.go:414] "containerMap: removing container" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerName="cleanup" Nov 02 16:00:17.452147 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.452140 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerName="cleanup" Nov 02 16:00:17.452310 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.452198 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerName="cleanup" Nov 02 16:00:17.452310 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.452208 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerName="cleanup" Nov 02 16:00:17.452310 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.452216 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" containerName="cleanup" Nov 02 16:00:17.456474 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.456447 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:17.486155 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.486120 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krlp2\" (UniqueName: \"kubernetes.io/projected/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-kube-api-access-krlp2\") pod \"maas-api-revoke-sub-d576e68b-gd2t4\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:17.486368 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.486253 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gd2t4\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:17.486368 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.486281 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gd2t4\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:17.514574 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.514537 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4"] Nov 02 16:00:17.555778 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.555732 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:00:17.555778 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.555768 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:00:17.556047 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.555750 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 16:00:17.556047 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.555745 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:00:17.587151 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.587115 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gd2t4\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:17.587381 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.587159 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gd2t4\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:17.587381 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.587272 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krlp2\" (UniqueName: \"kubernetes.io/projected/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-kube-api-access-krlp2\") pod \"maas-api-revoke-sub-d576e68b-gd2t4\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:17.587913 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.587893 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gd2t4\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:17.589664 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.589641 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gd2t4\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:17.658091 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.658052 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krlp2\" (UniqueName: \"kubernetes.io/projected/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-kube-api-access-krlp2\") pod \"maas-api-revoke-sub-d576e68b-gd2t4\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:17.770023 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.769932 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:17.923680 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:17.923650 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1c626d-1d1d-4a54-a175-83c56371e8cf" path="/var/lib/kubelet/pods/fc1c626d-1d1d-4a54-a175-83c56371e8cf/volumes" Nov 02 16:00:18.137183 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:18.137153 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4"] Nov 02 16:00:18.142854 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:00:18.142828 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef599cbf_e0bf_4fdb_b1ca_77457a46b34b.slice/crio-88b09985e33919411007fb7ff00b5167354ed96ae2a5ed1656bca039a7a7fbc4 WatchSource:0}: Error finding container 88b09985e33919411007fb7ff00b5167354ed96ae2a5ed1656bca039a7a7fbc4: Status 404 returned error can't find the container with id 88b09985e33919411007fb7ff00b5167354ed96ae2a5ed1656bca039a7a7fbc4 Nov 02 16:00:18.780372 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:18.780294 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" event={"ID":"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b","Type":"ContainerStarted","Data":"289b9c03f64ac6fdd278d1fd729a9c1cc4e497fc0a11b9d5abf5c62dfc363fad"} Nov 02 16:00:18.780372 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:18.780343 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" event={"ID":"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b","Type":"ContainerStarted","Data":"88b09985e33919411007fb7ff00b5167354ed96ae2a5ed1656bca039a7a7fbc4"} Nov 02 16:00:18.891490 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:18.891436 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" podStartSLOduration=1.891420648 podStartE2EDuration="1.891420648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:00:18.889283321 +0000 UTC m=+3721.092161446" watchObservedRunningTime="2026-11-02 16:00:18.891420648 +0000 UTC m=+3721.094298705" Nov 02 16:00:19.786969 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:19.786934 5476 generic.go:356] "Generic (PLEG): container finished" podID="ef599cbf-e0bf-4fdb-b1ca-77457a46b34b" containerID="289b9c03f64ac6fdd278d1fd729a9c1cc4e497fc0a11b9d5abf5c62dfc363fad" exitCode=0 Nov 02 16:00:19.787401 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:19.786976 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" event={"ID":"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b","Type":"ContainerDied","Data":"289b9c03f64ac6fdd278d1fd729a9c1cc4e497fc0a11b9d5abf5c62dfc363fad"} Nov 02 16:00:20.065207 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:20.065126 5476 scope.go:122] "RemoveContainer" containerID="587afe38dc115f3128ea9d99db00011bc33f560fb2bd8ec690c345cc0ccd8649" Nov 02 16:00:20.074115 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:20.074096 5476 scope.go:122] "RemoveContainer" containerID="b4f293d6015e99de659511be643306d3dc27c33c7c2a6268534868aa810d282d" Nov 02 16:00:20.145585 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:00:20.145549 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:00:20.921391 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:20.921369 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:21.024555 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.024527 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-openshift-ca\") pod \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " Nov 02 16:00:21.024845 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.024568 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-kube-api-access-krlp2\" (UniqueName: \"kubernetes.io/projected/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-kube-api-access-krlp2\") pod \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " Nov 02 16:00:21.024845 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.024588 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-xks-ca\") pod \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\" (UID: \"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b\") " Nov 02 16:00:21.024985 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.024897 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-openshift-ca" pod "ef599cbf-e0bf-4fdb-b1ca-77457a46b34b" (UID: "ef599cbf-e0bf-4fdb-b1ca-77457a46b34b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:00:21.026677 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.026652 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-xks-ca" pod "ef599cbf-e0bf-4fdb-b1ca-77457a46b34b" (UID: "ef599cbf-e0bf-4fdb-b1ca-77457a46b34b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:00:21.026778 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.026757 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-kube-api-access-krlp2" pod "ef599cbf-e0bf-4fdb-b1ca-77457a46b34b" (UID: "ef599cbf-e0bf-4fdb-b1ca-77457a46b34b"). InnerVolumeSpecName "kube-api-access-krlp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:00:21.126087 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.126060 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:00:21.126087 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.126085 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krlp2\" (UniqueName: \"kubernetes.io/projected/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-kube-api-access-krlp2\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:00:21.126279 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.126095 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:00:21.799897 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.799862 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" Nov 02 16:00:21.799897 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.799885 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4" event={"ID":"ef599cbf-e0bf-4fdb-b1ca-77457a46b34b","Type":"ContainerDied","Data":"88b09985e33919411007fb7ff00b5167354ed96ae2a5ed1656bca039a7a7fbc4"} Nov 02 16:00:21.800149 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:21.799964 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88b09985e33919411007fb7ff00b5167354ed96ae2a5ed1656bca039a7a7fbc4" Nov 02 16:00:26.838176 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:26.838108 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" event={"ID":"a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29","Type":"ContainerStarted","Data":"44723fa1fcee38658c1b0f6e439330e6bfd6e0ed41cea6721c4336e717fa7573"} Nov 02 16:00:26.948016 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:26.947953 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-k4hl6" podStartSLOduration=2.504744069 podStartE2EDuration="14.947940338s" totalImagesPullingTime="12.443196269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:00:26.946245329 +0000 UTC m=+3729.149123386" watchObservedRunningTime="2026-11-02 16:00:26.947940338 +0000 UTC m=+3729.150818397" Nov 02 16:00:30.197768 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:30.197736 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:00:39.389046 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:39.389007 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:00:46.366005 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.365957 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw"] Nov 02 16:00:46.366706 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.366685 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ef599cbf-e0bf-4fdb-b1ca-77457a46b34b" containerName="revoke-keys" Nov 02 16:00:46.366761 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.366710 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef599cbf-e0bf-4fdb-b1ca-77457a46b34b" containerName="revoke-keys" Nov 02 16:00:46.366884 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.366873 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef599cbf-e0bf-4fdb-b1ca-77457a46b34b" containerName="revoke-keys" Nov 02 16:00:46.374125 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.374101 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:46.425366 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.425334 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw"] Nov 02 16:00:46.425702 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.425681 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lz4pw\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:46.425878 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.425856 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lz4pw\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:46.425958 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.425902 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkxz\" (UniqueName: \"kubernetes.io/projected/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-kube-api-access-8mkxz\") pod \"maas-api-revoke-sub-d2bc93bc-lz4pw\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:46.468333 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.468305 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:00:46.475408 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.468767 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:00:46.475408 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.471671 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 16:00:46.475408 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.471737 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:00:46.526890 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.526861 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lz4pw\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:46.527076 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.526894 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkxz\" (UniqueName: \"kubernetes.io/projected/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-kube-api-access-8mkxz\") pod \"maas-api-revoke-sub-d2bc93bc-lz4pw\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:46.527076 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.526928 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lz4pw\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:46.527595 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.527570 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lz4pw\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:46.529261 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.529243 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-lz4pw\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:46.584803 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.584779 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkxz\" (UniqueName: \"kubernetes.io/projected/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-kube-api-access-8mkxz\") pod \"maas-api-revoke-sub-d2bc93bc-lz4pw\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:46.691418 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:46.691349 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:47.063111 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:47.063032 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw"] Nov 02 16:00:47.070005 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:00:47.069956 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbf7ae0_690a_4ba8_a2f7_fbc61e165545.slice/crio-9cdbee4bc804e1dce47740f3d94e16e2bc5843f8e380962749398b24ba67b21d WatchSource:0}: Error finding container 9cdbee4bc804e1dce47740f3d94e16e2bc5843f8e380962749398b24ba67b21d: Status 404 returned error can't find the container with id 9cdbee4bc804e1dce47740f3d94e16e2bc5843f8e380962749398b24ba67b21d Nov 02 16:00:47.969498 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:47.969467 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" event={"ID":"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545","Type":"ContainerStarted","Data":"68b6858d2ca4b5fd09dfb2c2b7db0b473493f4ec7e31db465f87347e9e7fa0ce"} Nov 02 16:00:47.969985 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:47.969521 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" event={"ID":"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545","Type":"ContainerStarted","Data":"9cdbee4bc804e1dce47740f3d94e16e2bc5843f8e380962749398b24ba67b21d"} Nov 02 16:00:48.078897 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:48.078827 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" podStartSLOduration=2.078809012 podStartE2EDuration="2.078809012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:00:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:00:48.078391637 +0000 UTC m=+3750.281269691" watchObservedRunningTime="2026-11-02 16:00:48.078809012 +0000 UTC m=+3750.281687078" Nov 02 16:00:48.976733 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:48.976698 5476 generic.go:356] "Generic (PLEG): container finished" podID="6bbf7ae0-690a-4ba8-a2f7-fbc61e165545" containerID="68b6858d2ca4b5fd09dfb2c2b7db0b473493f4ec7e31db465f87347e9e7fa0ce" exitCode=0 Nov 02 16:00:48.977336 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:48.976735 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" event={"ID":"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545","Type":"ContainerDied","Data":"68b6858d2ca4b5fd09dfb2c2b7db0b473493f4ec7e31db465f87347e9e7fa0ce"} Nov 02 16:00:50.120766 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.120743 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:50.144740 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:00:50.144709 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:00:50.162093 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.162071 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-xks-ca\") pod \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " Nov 02 16:00:50.162237 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.162141 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-kube-api-access-8mkxz\" (UniqueName: \"kubernetes.io/projected/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-kube-api-access-8mkxz\") pod \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " Nov 02 16:00:50.162237 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.162184 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-openshift-ca\") pod \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\" (UID: \"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545\") " Nov 02 16:00:50.162560 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.162532 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-openshift-ca" pod "6bbf7ae0-690a-4ba8-a2f7-fbc61e165545" (UID: "6bbf7ae0-690a-4ba8-a2f7-fbc61e165545"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:00:50.164117 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.164092 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-kube-api-access-8mkxz" pod "6bbf7ae0-690a-4ba8-a2f7-fbc61e165545" (UID: "6bbf7ae0-690a-4ba8-a2f7-fbc61e165545"). InnerVolumeSpecName "kube-api-access-8mkxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:00:50.164205 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.164094 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-xks-ca" pod "6bbf7ae0-690a-4ba8-a2f7-fbc61e165545" (UID: "6bbf7ae0-690a-4ba8-a2f7-fbc61e165545"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:00:50.264114 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.264042 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mkxz\" (UniqueName: \"kubernetes.io/projected/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-kube-api-access-8mkxz\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:00:50.264114 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.264079 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:00:50.264114 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.264095 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:00:50.988908 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.988875 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" Nov 02 16:00:50.988908 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.988886 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw" event={"ID":"6bbf7ae0-690a-4ba8-a2f7-fbc61e165545","Type":"ContainerDied","Data":"9cdbee4bc804e1dce47740f3d94e16e2bc5843f8e380962749398b24ba67b21d"} Nov 02 16:00:50.989152 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:50.988928 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cdbee4bc804e1dce47740f3d94e16e2bc5843f8e380962749398b24ba67b21d" Nov 02 16:00:53.048341 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:53.048303 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5"] Nov 02 16:00:53.095381 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:53.095347 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-btlq5"] Nov 02 16:00:53.920606 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:53.920571 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f73102-ec90-4c11-94ab-ff616719bc02" path="/var/lib/kubelet/pods/08f73102-ec90-4c11-94ab-ff616719bc02/volumes" Nov 02 16:00:59.149084 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:00:59.149048 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:01:04.133301 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:04.133267 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452"] Nov 02 16:01:04.178542 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:04.178514 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-b9452"] Nov 02 16:01:05.921436 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:05.921402 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c228fa0a-24ca-410e-90ff-3fa2b82000e9" path="/var/lib/kubelet/pods/c228fa0a-24ca-410e-90ff-3fa2b82000e9/volumes" Nov 02 16:01:09.497114 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.497081 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp"] Nov 02 16:01:09.497494 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.497473 5476 cpu_manager.go:414] "containerMap: removing container" podUID="6bbf7ae0-690a-4ba8-a2f7-fbc61e165545" containerName="revoke-keys" Nov 02 16:01:09.497494 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.497485 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbf7ae0-690a-4ba8-a2f7-fbc61e165545" containerName="revoke-keys" Nov 02 16:01:09.497610 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.497599 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbf7ae0-690a-4ba8-a2f7-fbc61e165545" containerName="revoke-keys" Nov 02 16:01:09.503762 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.503738 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:09.601358 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.601322 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-1d7f44-serving-cert\"" Nov 02 16:01:09.601528 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.601386 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 16:01:09.601528 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.601419 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:01:09.601528 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.601419 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wj8b6\"" Nov 02 16:01:09.601528 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.601470 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:01:09.603595 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.603573 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp"] Nov 02 16:01:09.644569 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.644538 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:09.644569 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.644568 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qrl\" (UniqueName: \"kubernetes.io/projected/231d3607-3aab-4122-8e6a-61ccb4f45a06-kube-api-access-m5qrl\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:09.644789 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.644596 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:09.644789 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.644666 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/231d3607-3aab-4122-8e6a-61ccb4f45a06-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:09.746242 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.746198 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:09.746467 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.746252 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qrl\" (UniqueName: \"kubernetes.io/projected/231d3607-3aab-4122-8e6a-61ccb4f45a06-kube-api-access-m5qrl\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:09.746467 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.746335 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:09.746467 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.746394 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/231d3607-3aab-4122-8e6a-61ccb4f45a06-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:09.746756 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:01:09.746584 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-serving-cert: secret "maas-api-e2e-worker-single-main-1d7f44-serving-cert" not found Nov 02 16:01:09.746756 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:01:09.746685 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-maas-api-tls podName:231d3607-3aab-4122-8e6a-61ccb4f45a06 nodeName:}" failed. No retries permitted until 2026-11-02 16:01:10.246659846 +0000 UTC m=+3772.449537901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-maas-api-tls") pod "maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" (UID: "231d3607-3aab-4122-8e6a-61ccb4f45a06") : secret "maas-api-e2e-worker-single-main-1d7f44-serving-cert" not found Nov 02 16:01:09.747361 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.747304 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/231d3607-3aab-4122-8e6a-61ccb4f45a06-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:09.749197 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.749166 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-metrics-certs\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:09.806401 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:09.806369 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qrl\" (UniqueName: \"kubernetes.io/projected/231d3607-3aab-4122-8e6a-61ccb4f45a06-kube-api-access-m5qrl\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:10.250655 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:10.250625 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:10.252959 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:10.252930 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:10.418234 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:10.418197 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:10.793546 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:10.793512 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp"] Nov 02 16:01:10.799693 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:01:10.799663 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231d3607_3aab_4122_8e6a_61ccb4f45a06.slice/crio-43db2bad3f0a2e1068f19a7d026a01befb0e11fd423345d9d6088a1d0e83b837 WatchSource:0}: Error finding container 43db2bad3f0a2e1068f19a7d026a01befb0e11fd423345d9d6088a1d0e83b837: Status 404 returned error can't find the container with id 43db2bad3f0a2e1068f19a7d026a01befb0e11fd423345d9d6088a1d0e83b837 Nov 02 16:01:11.098121 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:11.098032 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" event={"ID":"231d3607-3aab-4122-8e6a-61ccb4f45a06","Type":"ContainerStarted","Data":"43db2bad3f0a2e1068f19a7d026a01befb0e11fd423345d9d6088a1d0e83b837"} Nov 02 16:01:12.105682 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:12.105651 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" event={"ID":"231d3607-3aab-4122-8e6a-61ccb4f45a06","Type":"ContainerStarted","Data":"a3dba9ef0e02654a61678b29f5625617cfb1ecfc8cb13a5c189d54b1a0cd8d22"} Nov 02 16:01:12.106083 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:12.105830 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:12.218422 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:12.218365 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" podStartSLOduration=2.263158776 podStartE2EDuration="3.218347303s" totalImagesPullingTime="955.188527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:01:12.217706758 +0000 UTC m=+3774.420584819" watchObservedRunningTime="2026-11-02 16:01:12.218347303 +0000 UTC m=+3774.421225364" Nov 02 16:01:17.300708 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:17.300674 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:01:18.115740 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:18.115714 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:01:20.109142 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:20.109110 5476 scope.go:122] "RemoveContainer" containerID="55504cbef51913245abca164afd6e5ee20e0ff292de9abb319c6d6fb755f8fb8" Nov 02 16:01:20.118952 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:20.118929 5476 scope.go:122] "RemoveContainer" containerID="e8218418a18fe57307a67d4dd6ee4f6789796942d325c2ec5846ff3949ee8acc" Nov 02 16:01:20.129345 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:20.129328 5476 scope.go:122] "RemoveContainer" containerID="4be58ce97f86e15f2f807131f412a655b385b238de92e291c507ed53f0ba1e5c" Nov 02 16:01:20.149968 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:01:20.149933 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:01:36.683735 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:01:36.683645 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:01:50.146527 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:01:50.146485 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:02:03.971060 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:03.971025 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:02:20.148476 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:02:20.148430 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:02:20.161223 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:20.161196 5476 scope.go:122] "RemoveContainer" containerID="1af9207bd6b1015075a60883466675ea167977ca684106d56d60a6db3349bda6" Nov 02 16:02:20.171814 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:20.171792 5476 scope.go:122] "RemoveContainer" containerID="e5874212bcb0d74c58bd92ecb47ef483f57b1165d3a5237755285cf83941a3b7" Nov 02 16:02:20.180473 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:20.180449 5476 scope.go:122] "RemoveContainer" containerID="f28d9f5fc4c8e5768076349f5acbc58e1a19bc388457d5a1055d7f54a5e3a0a8" Nov 02 16:02:30.047941 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:30.047907 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:02:43.411145 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:43.411106 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp"] Nov 02 16:02:43.411604 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:43.411335 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" podUID="231d3607-3aab-4122-8e6a-61ccb4f45a06" containerName="maas-api" containerID="cri-o://a3dba9ef0e02654a61678b29f5625617cfb1ecfc8cb13a5c189d54b1a0cd8d22" gracePeriod=30 Nov 02 16:02:44.631589 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.631560 5476 generic.go:356] "Generic (PLEG): container finished" podID="231d3607-3aab-4122-8e6a-61ccb4f45a06" containerID="a3dba9ef0e02654a61678b29f5625617cfb1ecfc8cb13a5c189d54b1a0cd8d22" exitCode=0 Nov 02 16:02:44.631935 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.631633 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" event={"ID":"231d3607-3aab-4122-8e6a-61ccb4f45a06","Type":"ContainerDied","Data":"a3dba9ef0e02654a61678b29f5625617cfb1ecfc8cb13a5c189d54b1a0cd8d22"} Nov 02 16:02:44.658755 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.658734 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:02:44.667104 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.667084 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231d3607-3aab-4122-8e6a-61ccb4f45a06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/231d3607-3aab-4122-8e6a-61ccb4f45a06-openshift-service-ca\") pod \"231d3607-3aab-4122-8e6a-61ccb4f45a06\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " Nov 02 16:02:44.667217 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.667131 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-metrics-certs\") pod \"231d3607-3aab-4122-8e6a-61ccb4f45a06\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " Nov 02 16:02:44.667217 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.667188 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-maas-api-tls\") pod \"231d3607-3aab-4122-8e6a-61ccb4f45a06\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " Nov 02 16:02:44.667303 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.667230 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231d3607-3aab-4122-8e6a-61ccb4f45a06-kube-api-access-m5qrl\" (UniqueName: \"kubernetes.io/projected/231d3607-3aab-4122-8e6a-61ccb4f45a06-kube-api-access-m5qrl\") pod \"231d3607-3aab-4122-8e6a-61ccb4f45a06\" (UID: \"231d3607-3aab-4122-8e6a-61ccb4f45a06\") " Nov 02 16:02:44.667488 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.667465 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231d3607-3aab-4122-8e6a-61ccb4f45a06-openshift-service-ca" pod "231d3607-3aab-4122-8e6a-61ccb4f45a06" (UID: "231d3607-3aab-4122-8e6a-61ccb4f45a06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:02:44.667590 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.667536 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/231d3607-3aab-4122-8e6a-61ccb4f45a06-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:44.669242 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.669210 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-metrics-certs" pod "231d3607-3aab-4122-8e6a-61ccb4f45a06" (UID: "231d3607-3aab-4122-8e6a-61ccb4f45a06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:02:44.669242 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.669231 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-maas-api-tls" pod "231d3607-3aab-4122-8e6a-61ccb4f45a06" (UID: "231d3607-3aab-4122-8e6a-61ccb4f45a06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:02:44.669422 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.669296 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231d3607-3aab-4122-8e6a-61ccb4f45a06-kube-api-access-m5qrl" pod "231d3607-3aab-4122-8e6a-61ccb4f45a06" (UID: "231d3607-3aab-4122-8e6a-61ccb4f45a06"). InnerVolumeSpecName "kube-api-access-m5qrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:02:44.768285 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.768223 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:44.768285 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.768246 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/231d3607-3aab-4122-8e6a-61ccb4f45a06-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:44.768285 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:44.768257 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qrl\" (UniqueName: \"kubernetes.io/projected/231d3607-3aab-4122-8e6a-61ccb4f45a06-kube-api-access-m5qrl\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:45.638525 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:45.638492 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" Nov 02 16:02:45.638525 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:45.638508 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp" event={"ID":"231d3607-3aab-4122-8e6a-61ccb4f45a06","Type":"ContainerDied","Data":"43db2bad3f0a2e1068f19a7d026a01befb0e11fd423345d9d6088a1d0e83b837"} Nov 02 16:02:45.639039 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:45.638574 5476 scope.go:122] "RemoveContainer" containerID="a3dba9ef0e02654a61678b29f5625617cfb1ecfc8cb13a5c189d54b1a0cd8d22" Nov 02 16:02:45.791747 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:45.791718 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp"] Nov 02 16:02:45.837594 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:45.837567 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-1d7f44-5f998c4574-4p2bp"] Nov 02 16:02:45.927578 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:45.927455 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231d3607-3aab-4122-8e6a-61ccb4f45a06" path="/var/lib/kubelet/pods/231d3607-3aab-4122-8e6a-61ccb4f45a06/volumes" Nov 02 16:02:47.991291 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:02:47.991256 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:02:50.145816 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:02:50.145781 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:03:16.668816 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:16.668783 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:03:18.861698 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:18.861666 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 16:03:18.955511 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:18.955460 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 16:03:20.146030 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:03:20.145974 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:03:33.046612 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:33.046579 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8"] Nov 02 16:03:33.092348 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:33.092318 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-56lw8"] Nov 02 16:03:33.921346 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:33.921312 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5a5ee7-a423-40fa-b0d5-5e8817b753c3" path="/var/lib/kubelet/pods/ad5a5ee7-a423-40fa-b0d5-5e8817b753c3/volumes" Nov 02 16:03:43.154047 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.154013 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll"] Nov 02 16:03:43.154560 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.154471 5476 cpu_manager.go:414] "containerMap: removing container" podUID="231d3607-3aab-4122-8e6a-61ccb4f45a06" containerName="maas-api" Nov 02 16:03:43.154560 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.154487 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231d3607-3aab-4122-8e6a-61ccb4f45a06" containerName="maas-api" Nov 02 16:03:43.154655 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.154573 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="231d3607-3aab-4122-8e6a-61ccb4f45a06" containerName="maas-api" Nov 02 16:03:43.157722 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.157707 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.252582 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.252550 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d5201e83-kserve-self-signed-certs\"" Nov 02 16:03:43.254957 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.254934 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll"] Nov 02 16:03:43.276783 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.276758 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-model-cache\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.276871 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.276858 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-home\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.276911 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.276898 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-dshm\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.276950 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.276912 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tmp-dir\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.276950 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.276932 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5hh7\" (UniqueName: \"kubernetes.io/projected/57316be9-fcf2-4c32-b083-a5ef90e72fb4-kube-api-access-m5hh7\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.277092 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.277073 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tls-certs\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.378252 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.378219 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-model-cache\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.378460 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.378290 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-home\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.378460 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.378326 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-dshm\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.378460 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.378341 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tmp-dir\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.378460 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.378360 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5hh7\" (UniqueName: \"kubernetes.io/projected/57316be9-fcf2-4c32-b083-a5ef90e72fb4-kube-api-access-m5hh7\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.378460 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.378395 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tls-certs\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.378758 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.378684 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-model-cache\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.378758 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.378718 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tmp-dir\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.378841 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.378781 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-home\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.380598 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.380572 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-dshm\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.380844 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.380828 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tls-certs\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.436180 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.436120 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5hh7\" (UniqueName: \"kubernetes.io/projected/57316be9-fcf2-4c32-b083-a5ef90e72fb4-kube-api-access-m5hh7\") pod \"e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.468977 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.468951 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:43.845141 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.845061 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll"] Nov 02 16:03:43.850792 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:03:43.850758 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57316be9_fcf2_4c32_b083_a5ef90e72fb4.slice/crio-4039e18bc0d98b7c0660741f73ffa8e9261873bf0fd47aa3e388b4f6568be5d6 WatchSource:0}: Error finding container 4039e18bc0d98b7c0660741f73ffa8e9261873bf0fd47aa3e388b4f6568be5d6: Status 404 returned error can't find the container with id 4039e18bc0d98b7c0660741f73ffa8e9261873bf0fd47aa3e388b4f6568be5d6 Nov 02 16:03:43.974686 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.974656 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" event={"ID":"57316be9-fcf2-4c32-b083-a5ef90e72fb4","Type":"ContainerStarted","Data":"90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744"} Nov 02 16:03:43.974850 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:43.974696 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" event={"ID":"57316be9-fcf2-4c32-b083-a5ef90e72fb4","Type":"ContainerStarted","Data":"4039e18bc0d98b7c0660741f73ffa8e9261873bf0fd47aa3e388b4f6568be5d6"} Nov 02 16:03:44.084073 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:44.084024 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" podStartSLOduration=1.083987923 podStartE2EDuration="1.083987923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:03:44.083462276 +0000 UTC m=+3926.286340335" watchObservedRunningTime="2026-11-02 16:03:44.083987923 +0000 UTC m=+3926.286865983" Nov 02 16:03:44.469352 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:44.469321 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:47.683384 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:47.683343 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:03:50.164510 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:03:50.164478 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:03:50.644815 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:50.644128 5476 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kk2m9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 16:03:50.644815 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:50.644232 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kk2m9" podUID="afd13c1e-9b91-4be5-92ed-ceea14f0203c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 16:03:53.469818 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:53.469782 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:53.482169 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:53.482141 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:03:54.068430 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:03:54.068400 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:04:15.713222 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.713189 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f"] Nov 02 16:04:15.720371 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.720335 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:15.770579 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.770546 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:04:15.816642 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.816613 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 16:04:15.816821 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.816779 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:04:15.816894 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.816843 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:04:15.818990 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.818968 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f"] Nov 02 16:04:15.897456 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.897426 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95dj2\" (UniqueName: \"kubernetes.io/projected/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-kube-api-access-95dj2\") pod \"maas-api-revoke-sub-452b6d7b-qqk8f\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:15.897456 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.897460 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qqk8f\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:15.897705 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.897609 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qqk8f\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:15.999115 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.999042 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qqk8f\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:15.999264 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.999129 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95dj2\" (UniqueName: \"kubernetes.io/projected/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-kube-api-access-95dj2\") pod \"maas-api-revoke-sub-452b6d7b-qqk8f\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:15.999264 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.999153 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qqk8f\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:15.999680 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:15.999652 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qqk8f\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:16.001393 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:16.001373 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qqk8f\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:16.060568 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:16.060534 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95dj2\" (UniqueName: \"kubernetes.io/projected/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-kube-api-access-95dj2\") pod \"maas-api-revoke-sub-452b6d7b-qqk8f\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:16.335054 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:16.334964 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:16.703640 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:16.703610 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f"] Nov 02 16:04:16.709595 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:04:16.709561 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6d3822_ee5e_4b97_bc87_0dc1c15e89d0.slice/crio-627c639e00c8bf855ac3dc66b8d93746935ba564cc9750632388c792a07d881b WatchSource:0}: Error finding container 627c639e00c8bf855ac3dc66b8d93746935ba564cc9750632388c792a07d881b: Status 404 returned error can't find the container with id 627c639e00c8bf855ac3dc66b8d93746935ba564cc9750632388c792a07d881b Nov 02 16:04:16.966056 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:16.965964 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:04:17.179559 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:17.179519 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" event={"ID":"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0","Type":"ContainerStarted","Data":"96e04ab9196c498367ba98267719a03ede9353b766984cfd6e44c82dc1f38926"} Nov 02 16:04:17.179734 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:17.179586 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" event={"ID":"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0","Type":"ContainerStarted","Data":"627c639e00c8bf855ac3dc66b8d93746935ba564cc9750632388c792a07d881b"} Nov 02 16:04:17.291983 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:17.291890 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" podStartSLOduration=2.291876269 podStartE2EDuration="2.291876269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:04:17.289326721 +0000 UTC m=+3959.492204782" watchObservedRunningTime="2026-11-02 16:04:17.291876269 +0000 UTC m=+3959.494754328" Nov 02 16:04:18.187920 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:18.187882 5476 generic.go:356] "Generic (PLEG): container finished" podID="ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0" containerID="96e04ab9196c498367ba98267719a03ede9353b766984cfd6e44c82dc1f38926" exitCode=0 Nov 02 16:04:18.188362 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:18.187966 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" event={"ID":"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0","Type":"ContainerDied","Data":"96e04ab9196c498367ba98267719a03ede9353b766984cfd6e44c82dc1f38926"} Nov 02 16:04:19.557610 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:19.557292 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:19.655225 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:19.655189 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-openshift-ca\") pod \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " Nov 02 16:04:19.655403 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:19.655239 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-xks-ca\") pod \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " Nov 02 16:04:19.655403 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:19.655334 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-kube-api-access-95dj2\" (UniqueName: \"kubernetes.io/projected/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-kube-api-access-95dj2\") pod \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\" (UID: \"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0\") " Nov 02 16:04:19.655599 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:19.655572 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-openshift-ca" pod "ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0" (UID: "ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:04:19.657373 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:19.657351 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-xks-ca" pod "ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0" (UID: "ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:04:19.657430 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:19.657411 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-kube-api-access-95dj2" pod "ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0" (UID: "ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0"). InnerVolumeSpecName "kube-api-access-95dj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:04:19.756360 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:19.756324 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95dj2\" (UniqueName: \"kubernetes.io/projected/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-kube-api-access-95dj2\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:19.756360 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:19.756353 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:19.756360 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:19.756364 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:20.165567 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:04:20.165527 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:04:20.246438 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:20.246349 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" event={"ID":"ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0","Type":"ContainerDied","Data":"627c639e00c8bf855ac3dc66b8d93746935ba564cc9750632388c792a07d881b"} Nov 02 16:04:20.254036 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:20.253085 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="627c639e00c8bf855ac3dc66b8d93746935ba564cc9750632388c792a07d881b" Nov 02 16:04:20.254036 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:20.246718 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f" Nov 02 16:04:31.604042 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:31.603949 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll"] Nov 02 16:04:32.358757 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:32.358728 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:04:36.306221 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.306187 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc"] Nov 02 16:04:36.306646 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.306631 5476 cpu_manager.go:414] "containerMap: removing container" podUID="ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0" containerName="revoke-keys" Nov 02 16:04:36.306712 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.306649 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0" containerName="revoke-keys" Nov 02 16:04:36.306752 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.306721 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0" containerName="revoke-keys" Nov 02 16:04:36.309944 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.309926 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:36.358924 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.358890 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:04:36.359129 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.359055 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:04:36.405317 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.405288 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:04:36.405527 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.405508 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 16:04:36.407533 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.407514 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc"] Nov 02 16:04:36.450923 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.450898 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtdk\" (UniqueName: \"kubernetes.io/projected/4780db8d-0d1d-41ff-8495-eef08187c8c9-kube-api-access-7rtdk\") pod \"maas-api-revoke-sub-3e31a506-vhtjc\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:36.451081 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.450931 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vhtjc\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:36.451128 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.451088 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vhtjc\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:36.552151 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.552120 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vhtjc\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:36.552311 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.552174 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtdk\" (UniqueName: \"kubernetes.io/projected/4780db8d-0d1d-41ff-8495-eef08187c8c9-kube-api-access-7rtdk\") pod \"maas-api-revoke-sub-3e31a506-vhtjc\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:36.552311 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.552203 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vhtjc\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:36.552802 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.552779 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-vhtjc\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:36.554452 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.554432 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-vhtjc\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:36.609949 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.609920 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtdk\" (UniqueName: \"kubernetes.io/projected/4780db8d-0d1d-41ff-8495-eef08187c8c9-kube-api-access-7rtdk\") pod \"maas-api-revoke-sub-3e31a506-vhtjc\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:36.622700 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.622673 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:36.992020 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:36.991925 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc"] Nov 02 16:04:36.997535 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:04:36.997510 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4780db8d_0d1d_41ff_8495_eef08187c8c9.slice/crio-35de7b1251b302c078e3d841835df5697cecb90ad7b4f4406c5968e1e140b99b WatchSource:0}: Error finding container 35de7b1251b302c078e3d841835df5697cecb90ad7b4f4406c5968e1e140b99b: Status 404 returned error can't find the container with id 35de7b1251b302c078e3d841835df5697cecb90ad7b4f4406c5968e1e140b99b Nov 02 16:04:37.361284 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:37.361252 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" event={"ID":"4780db8d-0d1d-41ff-8495-eef08187c8c9","Type":"ContainerStarted","Data":"d32748506a8442ba2796c26d93019163ff93546f45b31e3dfd13888112d70be6"} Nov 02 16:04:37.361619 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:37.361306 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" event={"ID":"4780db8d-0d1d-41ff-8495-eef08187c8c9","Type":"ContainerStarted","Data":"35de7b1251b302c078e3d841835df5697cecb90ad7b4f4406c5968e1e140b99b"} Nov 02 16:04:37.472283 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:37.472222 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" podStartSLOduration=1.4722077630000001 podStartE2EDuration="1.472207763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:04:37.470138377 +0000 UTC m=+3979.673016438" watchObservedRunningTime="2026-11-02 16:04:37.472207763 +0000 UTC m=+3979.675085820" Nov 02 16:04:38.368520 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:38.368480 5476 generic.go:356] "Generic (PLEG): container finished" podID="4780db8d-0d1d-41ff-8495-eef08187c8c9" containerID="d32748506a8442ba2796c26d93019163ff93546f45b31e3dfd13888112d70be6" exitCode=0 Nov 02 16:04:38.368970 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:38.368570 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" event={"ID":"4780db8d-0d1d-41ff-8495-eef08187c8c9","Type":"ContainerDied","Data":"d32748506a8442ba2796c26d93019163ff93546f45b31e3dfd13888112d70be6"} Nov 02 16:04:39.540128 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:39.540095 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:39.683058 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:39.682275 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4780db8d-0d1d-41ff-8495-eef08187c8c9-kube-api-access-7rtdk\" (UniqueName: \"kubernetes.io/projected/4780db8d-0d1d-41ff-8495-eef08187c8c9-kube-api-access-7rtdk\") pod \"4780db8d-0d1d-41ff-8495-eef08187c8c9\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " Nov 02 16:04:39.683058 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:39.682394 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-openshift-ca\") pod \"4780db8d-0d1d-41ff-8495-eef08187c8c9\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " Nov 02 16:04:39.683058 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:39.682443 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-xks-ca\") pod \"4780db8d-0d1d-41ff-8495-eef08187c8c9\" (UID: \"4780db8d-0d1d-41ff-8495-eef08187c8c9\") " Nov 02 16:04:39.683376 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:39.683081 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-openshift-ca" pod "4780db8d-0d1d-41ff-8495-eef08187c8c9" (UID: "4780db8d-0d1d-41ff-8495-eef08187c8c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:04:39.684960 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:39.684931 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-xks-ca" pod "4780db8d-0d1d-41ff-8495-eef08187c8c9" (UID: "4780db8d-0d1d-41ff-8495-eef08187c8c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:04:39.685163 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:39.685139 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4780db8d-0d1d-41ff-8495-eef08187c8c9-kube-api-access-7rtdk" pod "4780db8d-0d1d-41ff-8495-eef08187c8c9" (UID: "4780db8d-0d1d-41ff-8495-eef08187c8c9"). InnerVolumeSpecName "kube-api-access-7rtdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:04:39.784334 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:39.784298 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:39.784334 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:39.784338 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4780db8d-0d1d-41ff-8495-eef08187c8c9-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:39.784552 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:39.784354 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtdk\" (UniqueName: \"kubernetes.io/projected/4780db8d-0d1d-41ff-8495-eef08187c8c9-kube-api-access-7rtdk\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:40.412015 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:40.411807 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" event={"ID":"4780db8d-0d1d-41ff-8495-eef08187c8c9","Type":"ContainerDied","Data":"35de7b1251b302c078e3d841835df5697cecb90ad7b4f4406c5968e1e140b99b"} Nov 02 16:04:40.412015 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:40.411874 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35de7b1251b302c078e3d841835df5697cecb90ad7b4f4406c5968e1e140b99b" Nov 02 16:04:40.412015 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:40.411942 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc" Nov 02 16:04:43.942049 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:43.942017 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:04:46.623630 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.623572 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" podUID="57316be9-fcf2-4c32-b083-a5ef90e72fb4" containerName="main" containerID="cri-o://90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744" gracePeriod=45 Nov 02 16:04:46.874961 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.874907 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:04:46.886037 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886016 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-model-cache\") pod \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " Nov 02 16:04:46.886129 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886058 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tls-certs\" (UniqueName: \"kubernetes.io/secret/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tls-certs\") pod \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " Nov 02 16:04:46.886129 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886095 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57316be9-fcf2-4c32-b083-a5ef90e72fb4-kube-api-access-m5hh7\" (UniqueName: \"kubernetes.io/projected/57316be9-fcf2-4c32-b083-a5ef90e72fb4-kube-api-access-m5hh7\") pod \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " Nov 02 16:04:46.886129 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886116 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tmp-dir\") pod \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " Nov 02 16:04:46.886292 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886141 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-home\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-home\") pod \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " Nov 02 16:04:46.886292 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886240 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-model-cache" pod "57316be9-fcf2-4c32-b083-a5ef90e72fb4" (UID: "57316be9-fcf2-4c32-b083-a5ef90e72fb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:04:46.886292 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886257 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-dshm\") pod \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\" (UID: \"57316be9-fcf2-4c32-b083-a5ef90e72fb4\") " Nov 02 16:04:46.886462 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886424 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tmp-dir" pod "57316be9-fcf2-4c32-b083-a5ef90e72fb4" (UID: "57316be9-fcf2-4c32-b083-a5ef90e72fb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:04:46.886516 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886463 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-home" pod "57316be9-fcf2-4c32-b083-a5ef90e72fb4" (UID: "57316be9-fcf2-4c32-b083-a5ef90e72fb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:04:46.886647 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886626 5476 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-model-cache\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:46.886647 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886644 5476 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tmp-dir\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:46.886831 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.886654 5476 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-home\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:46.888299 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.888271 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-dshm" pod "57316be9-fcf2-4c32-b083-a5ef90e72fb4" (UID: "57316be9-fcf2-4c32-b083-a5ef90e72fb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:04:46.888404 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.888350 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57316be9-fcf2-4c32-b083-a5ef90e72fb4-kube-api-access-m5hh7" pod "57316be9-fcf2-4c32-b083-a5ef90e72fb4" (UID: "57316be9-fcf2-4c32-b083-a5ef90e72fb4"). InnerVolumeSpecName "kube-api-access-m5hh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:04:46.888404 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.888392 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tls-certs" pod "57316be9-fcf2-4c32-b083-a5ef90e72fb4" (UID: "57316be9-fcf2-4c32-b083-a5ef90e72fb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:04:46.987974 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.987938 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5hh7\" (UniqueName: \"kubernetes.io/projected/57316be9-fcf2-4c32-b083-a5ef90e72fb4-kube-api-access-m5hh7\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:46.987974 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.987972 5476 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57316be9-fcf2-4c32-b083-a5ef90e72fb4-dshm\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:46.988224 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:46.987983 5476 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57316be9-fcf2-4c32-b083-a5ef90e72fb4-tls-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:47.452961 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:47.452924 5476 generic.go:356] "Generic (PLEG): container finished" podID="57316be9-fcf2-4c32-b083-a5ef90e72fb4" containerID="90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744" exitCode=0 Nov 02 16:04:47.453245 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:47.452968 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" Nov 02 16:04:47.453245 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:47.452972 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" event={"ID":"57316be9-fcf2-4c32-b083-a5ef90e72fb4","Type":"ContainerDied","Data":"90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744"} Nov 02 16:04:47.453245 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:47.453048 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll" event={"ID":"57316be9-fcf2-4c32-b083-a5ef90e72fb4","Type":"ContainerDied","Data":"4039e18bc0d98b7c0660741f73ffa8e9261873bf0fd47aa3e388b4f6568be5d6"} Nov 02 16:04:47.453245 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:47.453077 5476 scope.go:122] "RemoveContainer" containerID="90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744" Nov 02 16:04:47.462906 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:47.462889 5476 scope.go:122] "RemoveContainer" containerID="90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744" Nov 02 16:04:47.463245 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:04:47.463225 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744\": container with ID starting with 90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744 not found: ID does not exist" containerID="90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744" Nov 02 16:04:47.463331 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:47.463257 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744"} err="failed to get container status \"90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744\": rpc error: code = NotFound desc = could not find container \"90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744\": container with ID starting with 90908d8678789014223c9c07e458a793eca5c2d129d7afd77e349cb74d77c744 not found: ID does not exist" Nov 02 16:04:47.608470 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:47.608435 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll"] Nov 02 16:04:47.654713 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:47.654679 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d5201e83-kserve-76cd5d9789-6zbll"] Nov 02 16:04:47.923237 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:47.923198 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57316be9-fcf2-4c32-b083-a5ef90e72fb4" path="/var/lib/kubelet/pods/57316be9-fcf2-4c32-b083-a5ef90e72fb4/volumes" Nov 02 16:04:50.145596 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:04:50.145561 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:04:54.971786 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:04:54.971753 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:05:02.619817 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.619783 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq"] Nov 02 16:05:02.620237 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.620184 5476 cpu_manager.go:414] "containerMap: removing container" podUID="57316be9-fcf2-4c32-b083-a5ef90e72fb4" containerName="main" Nov 02 16:05:02.620237 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.620196 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57316be9-fcf2-4c32-b083-a5ef90e72fb4" containerName="main" Nov 02 16:05:02.620237 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.620212 5476 cpu_manager.go:414] "containerMap: removing container" podUID="4780db8d-0d1d-41ff-8495-eef08187c8c9" containerName="revoke-keys" Nov 02 16:05:02.620237 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.620216 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4780db8d-0d1d-41ff-8495-eef08187c8c9" containerName="revoke-keys" Nov 02 16:05:02.620373 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.620292 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="57316be9-fcf2-4c32-b083-a5ef90e72fb4" containerName="main" Nov 02 16:05:02.620373 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.620302 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4780db8d-0d1d-41ff-8495-eef08187c8c9" containerName="revoke-keys" Nov 02 16:05:02.623523 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.623506 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:02.672736 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.672711 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:05:02.672736 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.672733 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:05:02.690093 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.690046 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa999e8-ws2bq\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:02.690271 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.690115 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvt6n\" (UniqueName: \"kubernetes.io/projected/9457f436-daff-49ea-8158-da6b5f218b3c-kube-api-access-kvt6n\") pod \"maas-api-revoke-sub-8fa999e8-ws2bq\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:02.690271 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.690178 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa999e8-ws2bq\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:02.719070 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.719045 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:05:02.719233 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.719129 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 16:05:02.721277 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.721257 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq"] Nov 02 16:05:02.791511 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.791480 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa999e8-ws2bq\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:02.791712 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.791526 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvt6n\" (UniqueName: \"kubernetes.io/projected/9457f436-daff-49ea-8158-da6b5f218b3c-kube-api-access-kvt6n\") pod \"maas-api-revoke-sub-8fa999e8-ws2bq\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:02.791712 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.791548 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa999e8-ws2bq\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:02.792184 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.792156 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa999e8-ws2bq\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:02.793857 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.793835 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa999e8-ws2bq\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:02.849214 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.849184 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvt6n\" (UniqueName: \"kubernetes.io/projected/9457f436-daff-49ea-8158-da6b5f218b3c-kube-api-access-kvt6n\") pod \"maas-api-revoke-sub-8fa999e8-ws2bq\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:02.936511 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:02.936436 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:03.306165 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:03.306082 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq"] Nov 02 16:05:03.311615 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:05:03.311575 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9457f436_daff_49ea_8158_da6b5f218b3c.slice/crio-02cb1735381bd649c5142d35a15956f55fc280741aaa3b5bfd839794a516f897 WatchSource:0}: Error finding container 02cb1735381bd649c5142d35a15956f55fc280741aaa3b5bfd839794a516f897: Status 404 returned error can't find the container with id 02cb1735381bd649c5142d35a15956f55fc280741aaa3b5bfd839794a516f897 Nov 02 16:05:03.584725 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:03.584693 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" event={"ID":"9457f436-daff-49ea-8158-da6b5f218b3c","Type":"ContainerStarted","Data":"02cb1735381bd649c5142d35a15956f55fc280741aaa3b5bfd839794a516f897"} Nov 02 16:05:04.592155 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:04.592062 5476 generic.go:356] "Generic (PLEG): container finished" podID="9457f436-daff-49ea-8158-da6b5f218b3c" containerID="6f63fc702777431796bc3b1fa75274c38267eb0405ae91e5ec4a98ef30d6dd3d" exitCode=0 Nov 02 16:05:04.592155 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:04.592115 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" event={"ID":"9457f436-daff-49ea-8158-da6b5f218b3c","Type":"ContainerDied","Data":"6f63fc702777431796bc3b1fa75274c38267eb0405ae91e5ec4a98ef30d6dd3d"} Nov 02 16:05:05.731342 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:05.731317 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:05.817463 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:05.817438 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9457f436-daff-49ea-8158-da6b5f218b3c-kube-api-access-kvt6n\" (UniqueName: \"kubernetes.io/projected/9457f436-daff-49ea-8158-da6b5f218b3c-kube-api-access-kvt6n\") pod \"9457f436-daff-49ea-8158-da6b5f218b3c\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " Nov 02 16:05:05.817611 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:05.817483 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-openshift-ca\") pod \"9457f436-daff-49ea-8158-da6b5f218b3c\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " Nov 02 16:05:05.817695 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:05.817673 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-xks-ca\") pod \"9457f436-daff-49ea-8158-da6b5f218b3c\" (UID: \"9457f436-daff-49ea-8158-da6b5f218b3c\") " Nov 02 16:05:05.817852 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:05.817828 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-openshift-ca" pod "9457f436-daff-49ea-8158-da6b5f218b3c" (UID: "9457f436-daff-49ea-8158-da6b5f218b3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:05:05.818042 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:05.818024 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-openshift-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:05:05.819530 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:05.819508 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9457f436-daff-49ea-8158-da6b5f218b3c-kube-api-access-kvt6n" pod "9457f436-daff-49ea-8158-da6b5f218b3c" (UID: "9457f436-daff-49ea-8158-da6b5f218b3c"). InnerVolumeSpecName "kube-api-access-kvt6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:05:05.819588 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:05.819550 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-xks-ca" pod "9457f436-daff-49ea-8158-da6b5f218b3c" (UID: "9457f436-daff-49ea-8158-da6b5f218b3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:05:05.918986 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:05.918953 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvt6n\" (UniqueName: \"kubernetes.io/projected/9457f436-daff-49ea-8158-da6b5f218b3c-kube-api-access-kvt6n\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:05:05.918986 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:05.918987 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9457f436-daff-49ea-8158-da6b5f218b3c-maas-api-xks-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:05:06.603522 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:06.603489 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" Nov 02 16:05:06.603522 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:06.603502 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq" event={"ID":"9457f436-daff-49ea-8158-da6b5f218b3c","Type":"ContainerDied","Data":"02cb1735381bd649c5142d35a15956f55fc280741aaa3b5bfd839794a516f897"} Nov 02 16:05:06.603736 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:06.603543 5476 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02cb1735381bd649c5142d35a15956f55fc280741aaa3b5bfd839794a516f897" Nov 02 16:05:15.371284 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:15.371249 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:05:15.932668 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:15.932634 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz"] Nov 02 16:05:15.933049 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:15.933035 5476 cpu_manager.go:414] "containerMap: removing container" podUID="9457f436-daff-49ea-8158-da6b5f218b3c" containerName="revoke-keys" Nov 02 16:05:15.933111 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:15.933049 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9457f436-daff-49ea-8158-da6b5f218b3c" containerName="revoke-keys" Nov 02 16:05:15.933151 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:15.933147 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="9457f436-daff-49ea-8158-da6b5f218b3c" containerName="revoke-keys" Nov 02 16:05:15.936553 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:15.936531 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.013397 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.013363 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-maas-api-tls\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.013569 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.013488 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70fa749c-b6b9-4849-a297-5c3692ccd58d-openshift-service-ca\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.013569 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.013515 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6xb\" (UniqueName: \"kubernetes.io/projected/70fa749c-b6b9-4849-a297-5c3692ccd58d-kube-api-access-7l6xb\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.013569 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.013560 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-metrics-certs\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.032078 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.032051 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wj8b6\"" Nov 02 16:05:16.032078 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.032067 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 16:05:16.032680 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.032660 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 16:05:16.032798 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.032782 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:05:16.032864 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.032832 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:05:16.035323 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.035304 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz"] Nov 02 16:05:16.114234 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.114199 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-maas-api-tls\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.114427 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.114278 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70fa749c-b6b9-4849-a297-5c3692ccd58d-openshift-service-ca\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.114427 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.114304 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6xb\" (UniqueName: \"kubernetes.io/projected/70fa749c-b6b9-4849-a297-5c3692ccd58d-kube-api-access-7l6xb\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.114427 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.114319 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-metrics-certs\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.115041 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.115012 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70fa749c-b6b9-4849-a297-5c3692ccd58d-openshift-service-ca\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.116643 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.116623 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-metrics-certs\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.116754 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.116675 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-maas-api-tls\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.172027 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.171973 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6xb\" (UniqueName: \"kubernetes.io/projected/70fa749c-b6b9-4849-a297-5c3692ccd58d-kube-api-access-7l6xb\") pod \"maas-api-74cdd4d67-s7nnz\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.249285 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.249201 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:16.619928 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.619894 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz"] Nov 02 16:05:16.625825 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:05:16.625800 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fa749c_b6b9_4849_a297_5c3692ccd58d.slice/crio-d4419646bd039ed45819a6b1e398677340ae83678774417df62645167881103c WatchSource:0}: Error finding container d4419646bd039ed45819a6b1e398677340ae83678774417df62645167881103c: Status 404 returned error can't find the container with id d4419646bd039ed45819a6b1e398677340ae83678774417df62645167881103c Nov 02 16:05:16.627145 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.627127 5476 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 16:05:16.659071 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:16.659036 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" event={"ID":"70fa749c-b6b9-4849-a297-5c3692ccd58d","Type":"ContainerStarted","Data":"d4419646bd039ed45819a6b1e398677340ae83678774417df62645167881103c"} Nov 02 16:05:18.671823 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:18.671782 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" event={"ID":"70fa749c-b6b9-4849-a297-5c3692ccd58d","Type":"ContainerStarted","Data":"bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1"} Nov 02 16:05:18.672363 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:18.671921 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:18.786900 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:18.786848 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" podStartSLOduration=2.823080398 podStartE2EDuration="3.786836113s" totalImagesPullingTime="963.755715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:05:18.78625149 +0000 UTC m=+4020.989129551" watchObservedRunningTime="2026-11-02 16:05:18.786836113 +0000 UTC m=+4020.989714173" Nov 02 16:05:20.145316 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:05:20.145278 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:05:20.266408 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:20.266375 5476 scope.go:122] "RemoveContainer" containerID="3a29dde030b9e750f7f68d62b2d439967594720ce3efdf3f9d538a99f8b86c18" Nov 02 16:05:21.046401 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:21.046370 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4"] Nov 02 16:05:21.092511 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:21.092481 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gd2t4"] Nov 02 16:05:21.921036 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:21.920988 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef599cbf-e0bf-4fdb-b1ca-77457a46b34b" path="/var/lib/kubelet/pods/ef599cbf-e0bf-4fdb-b1ca-77457a46b34b/volumes" Nov 02 16:05:22.763581 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.763551 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc"] Nov 02 16:05:22.768596 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.768573 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.818836 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.818801 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f22ac84e-openshift-default-dockercfg-742zt\"" Nov 02 16:05:22.871049 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.871014 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc"] Nov 02 16:05:22.884592 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.884565 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7gtx\" (UniqueName: \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-kube-api-access-v7gtx\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.884755 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.884601 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-podinfo\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.884755 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.884735 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-data\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.884847 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.884779 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-envoy\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.884847 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.884807 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istiod-ca-cert\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.884969 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.884948 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-token\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.885039 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.885025 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-certs\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.885144 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.885118 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-socket\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.885256 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.885161 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-credential-socket\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.985926 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.985895 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-token\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986416 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.985948 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-certs\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986416 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.985976 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-socket\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986416 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.986016 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-credential-socket\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986416 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.986076 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7gtx\" (UniqueName: \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-kube-api-access-v7gtx\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986416 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.986097 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-podinfo\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986416 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.986134 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-data\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986416 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.986161 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-envoy\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986416 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.986184 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istiod-ca-cert\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986759 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.986429 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-certs\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986759 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.986475 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-credential-socket\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986759 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.986497 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-socket\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986759 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.986630 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-data\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.986971 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.986953 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istiod-ca-cert\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.988435 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.988416 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-envoy\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:22.988683 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:22.988661 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-podinfo\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:23.043489 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:23.043423 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-token\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:23.043630 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:23.043588 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7gtx\" (UniqueName: \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-kube-api-access-v7gtx\") pod \"e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:23.081418 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:23.081391 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:23.504853 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:23.504809 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc"] Nov 02 16:05:23.511739 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:05:23.511703 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b9a109_b109_4d55_bbac_6f9b8b7bc1bb.slice/crio-376bb6c645bdd773461c6acf0db96bd1afbf56db5118dd345036da23226983aa WatchSource:0}: Error finding container 376bb6c645bdd773461c6acf0db96bd1afbf56db5118dd345036da23226983aa: Status 404 returned error can't find the container with id 376bb6c645bdd773461c6acf0db96bd1afbf56db5118dd345036da23226983aa Nov 02 16:05:23.516700 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:23.516668 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:05:23.516796 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:23.516729 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:05:23.516796 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:23.516760 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:05:23.700187 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:23.700149 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" event={"ID":"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb","Type":"ContainerStarted","Data":"17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e"} Nov 02 16:05:23.700393 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:23.700248 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" event={"ID":"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb","Type":"ContainerStarted","Data":"376bb6c645bdd773461c6acf0db96bd1afbf56db5118dd345036da23226983aa"} Nov 02 16:05:23.817372 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:23.817265 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" podStartSLOduration=1.817250365 podStartE2EDuration="1.817250365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:05:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:05:23.814612534 +0000 UTC m=+4026.017490598" watchObservedRunningTime="2026-11-02 16:05:23.817250365 +0000 UTC m=+4026.020128479" Nov 02 16:05:24.082375 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.082295 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:24.087195 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.087168 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:24.243983 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.243943 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz"] Nov 02 16:05:24.244249 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.244204 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" podUID="70fa749c-b6b9-4849-a297-5c3692ccd58d" containerName="maas-api" containerID="cri-o://bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1" gracePeriod=30 Nov 02 16:05:24.249213 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.249193 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:24.294084 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.294047 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr"] Nov 02 16:05:24.297653 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.297630 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.354619 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.354545 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr"] Nov 02 16:05:24.401613 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.401578 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4e57a0-9bee-4b65-9b96-934bb2f16430-openshift-service-ca\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.401778 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.401626 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4e57a0-9bee-4b65-9b96-934bb2f16430-maas-api-tls\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.401778 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.401644 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4e57a0-9bee-4b65-9b96-934bb2f16430-metrics-certs\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.401969 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.401939 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzzgv\" (UniqueName: \"kubernetes.io/projected/9c4e57a0-9bee-4b65-9b96-934bb2f16430-kube-api-access-jzzgv\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.497932 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.497905 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:24.503124 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.503105 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzzgv\" (UniqueName: \"kubernetes.io/projected/9c4e57a0-9bee-4b65-9b96-934bb2f16430-kube-api-access-jzzgv\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.503193 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.503133 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4e57a0-9bee-4b65-9b96-934bb2f16430-openshift-service-ca\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.503193 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.503161 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4e57a0-9bee-4b65-9b96-934bb2f16430-maas-api-tls\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.503360 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.503343 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4e57a0-9bee-4b65-9b96-934bb2f16430-metrics-certs\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.503860 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.503835 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4e57a0-9bee-4b65-9b96-934bb2f16430-openshift-service-ca\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.505649 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.505623 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4e57a0-9bee-4b65-9b96-934bb2f16430-maas-api-tls\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.505738 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.505665 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c4e57a0-9bee-4b65-9b96-934bb2f16430-metrics-certs\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.560642 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.560621 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzzgv\" (UniqueName: \"kubernetes.io/projected/9c4e57a0-9bee-4b65-9b96-934bb2f16430-kube-api-access-jzzgv\") pod \"maas-api-57678c84d7-ggjmr\" (UID: \"9c4e57a0-9bee-4b65-9b96-934bb2f16430\") " pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.604257 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.604228 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70fa749c-b6b9-4849-a297-5c3692ccd58d-kube-api-access-7l6xb\" (UniqueName: \"kubernetes.io/projected/70fa749c-b6b9-4849-a297-5c3692ccd58d-kube-api-access-7l6xb\") pod \"70fa749c-b6b9-4849-a297-5c3692ccd58d\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " Nov 02 16:05:24.604402 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.604310 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-metrics-certs\") pod \"70fa749c-b6b9-4849-a297-5c3692ccd58d\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " Nov 02 16:05:24.604402 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.604341 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70fa749c-b6b9-4849-a297-5c3692ccd58d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70fa749c-b6b9-4849-a297-5c3692ccd58d-openshift-service-ca\") pod \"70fa749c-b6b9-4849-a297-5c3692ccd58d\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " Nov 02 16:05:24.604402 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.604368 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-maas-api-tls\") pod \"70fa749c-b6b9-4849-a297-5c3692ccd58d\" (UID: \"70fa749c-b6b9-4849-a297-5c3692ccd58d\") " Nov 02 16:05:24.604712 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.604661 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70fa749c-b6b9-4849-a297-5c3692ccd58d-openshift-service-ca" pod "70fa749c-b6b9-4849-a297-5c3692ccd58d" (UID: "70fa749c-b6b9-4849-a297-5c3692ccd58d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:05:24.606244 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.606225 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fa749c-b6b9-4849-a297-5c3692ccd58d-kube-api-access-7l6xb" pod "70fa749c-b6b9-4849-a297-5c3692ccd58d" (UID: "70fa749c-b6b9-4849-a297-5c3692ccd58d"). InnerVolumeSpecName "kube-api-access-7l6xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:05:24.606328 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.606285 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-maas-api-tls" pod "70fa749c-b6b9-4849-a297-5c3692ccd58d" (UID: "70fa749c-b6b9-4849-a297-5c3692ccd58d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:05:24.606386 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.606338 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-metrics-certs" pod "70fa749c-b6b9-4849-a297-5c3692ccd58d" (UID: "70fa749c-b6b9-4849-a297-5c3692ccd58d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:05:24.629720 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.629699 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:24.705685 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.705649 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70fa749c-b6b9-4849-a297-5c3692ccd58d-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:05:24.705685 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.705683 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:05:24.705971 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.705699 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6xb\" (UniqueName: \"kubernetes.io/projected/70fa749c-b6b9-4849-a297-5c3692ccd58d-kube-api-access-7l6xb\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:05:24.705971 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.705716 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70fa749c-b6b9-4849-a297-5c3692ccd58d-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:05:24.707796 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.707768 5476 generic.go:356] "Generic (PLEG): container finished" podID="70fa749c-b6b9-4849-a297-5c3692ccd58d" containerID="bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1" exitCode=0 Nov 02 16:05:24.707918 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.707826 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" Nov 02 16:05:24.707918 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.707839 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" event={"ID":"70fa749c-b6b9-4849-a297-5c3692ccd58d","Type":"ContainerDied","Data":"bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1"} Nov 02 16:05:24.707918 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.707911 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz" event={"ID":"70fa749c-b6b9-4849-a297-5c3692ccd58d","Type":"ContainerDied","Data":"d4419646bd039ed45819a6b1e398677340ae83678774417df62645167881103c"} Nov 02 16:05:24.708093 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.707943 5476 scope.go:122] "RemoveContainer" containerID="bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1" Nov 02 16:05:24.708588 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.708347 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:24.709421 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.709402 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:24.719567 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.719544 5476 scope.go:122] "RemoveContainer" containerID="bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1" Nov 02 16:05:24.719852 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:05:24.719831 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1\": container with ID starting with bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1 not found: ID does not exist" containerID="bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1" Nov 02 16:05:24.719921 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.719860 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1"} err="failed to get container status \"bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1\": rpc error: code = NotFound desc = could not find container \"bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1\": container with ID starting with bb984224ac25aae39382da8b485a44a38031e230903c339e6c06d5316ed791a1 not found: ID does not exist" Nov 02 16:05:24.999234 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:24.999201 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz"] Nov 02 16:05:25.015781 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:25.015752 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr"] Nov 02 16:05:25.018017 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:25.017964 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74cdd4d67-s7nnz"] Nov 02 16:05:25.022191 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:05:25.022166 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4e57a0_9bee_4b65_9b96_934bb2f16430.slice/crio-89c24ddd3e165e3bb2322a4a7e770eb1c29b4f87132253efbe9f2b46af7a8195 WatchSource:0}: Error finding container 89c24ddd3e165e3bb2322a4a7e770eb1c29b4f87132253efbe9f2b46af7a8195: Status 404 returned error can't find the container with id 89c24ddd3e165e3bb2322a4a7e770eb1c29b4f87132253efbe9f2b46af7a8195 Nov 02 16:05:25.715488 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:25.715452 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" event={"ID":"9c4e57a0-9bee-4b65-9b96-934bb2f16430","Type":"ContainerStarted","Data":"89c24ddd3e165e3bb2322a4a7e770eb1c29b4f87132253efbe9f2b46af7a8195"} Nov 02 16:05:25.921088 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:25.921063 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70fa749c-b6b9-4849-a297-5c3692ccd58d" path="/var/lib/kubelet/pods/70fa749c-b6b9-4849-a297-5c3692ccd58d/volumes" Nov 02 16:05:26.722459 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:26.722420 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" event={"ID":"9c4e57a0-9bee-4b65-9b96-934bb2f16430","Type":"ContainerStarted","Data":"7af7a33563d7937865209a42277a4352470f96ad3f28e67e23ca87c992fa2710"} Nov 02 16:05:26.833263 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:26.833219 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" podStartSLOduration=1.960752265 podStartE2EDuration="2.833206184s" totalImagesPullingTime="872.453919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:05:26.832896698 +0000 UTC m=+4029.035774759" watchObservedRunningTime="2026-11-02 16:05:26.833206184 +0000 UTC m=+4029.036084244" Nov 02 16:05:27.727868 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:27.727837 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:33.739089 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:33.739062 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57678c84d7-ggjmr" Nov 02 16:05:38.903528 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:38.903498 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:05:48.849088 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:48.848988 5476 generic.go:356] "Generic (PLEG): container finished" podID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerID="17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e" exitCode=0 Nov 02 16:05:48.849088 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:48.849063 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" event={"ID":"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb","Type":"ContainerDied","Data":"17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e"} Nov 02 16:05:48.849651 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:48.849444 5476 scope.go:122] "RemoveContainer" containerID="17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e" Nov 02 16:05:48.850611 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:48.850585 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:05:48.850694 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:48.850648 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:05:48.850694 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:48.850683 5476 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 16:05:49.857271 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:49.857236 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" event={"ID":"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb","Type":"ContainerStarted","Data":"9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128"} Nov 02 16:05:50.093022 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:50.092380 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:50.096981 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:50.096952 5476 patch_prober.go:29] interesting pod/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 16:05:50.097178 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:50.097037 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 16:05:50.146284 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:05:50.146208 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:05:51.048918 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:51.048881 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw"] Nov 02 16:05:51.053386 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:51.053361 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-lz4pw"] Nov 02 16:05:51.088378 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:51.088346 5476 patch_prober.go:29] interesting pod/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 16:05:51.088636 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:51.088610 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 16:05:51.921909 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:51.921871 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bbf7ae0-690a-4ba8-a2f7-fbc61e165545" path="/var/lib/kubelet/pods/6bbf7ae0-690a-4ba8-a2f7-fbc61e165545/volumes" Nov 02 16:05:52.095455 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:52.095417 5476 patch_prober.go:29] interesting pod/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 16:05:52.096118 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:52.096080 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 16:05:53.090758 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:53.090720 5476 patch_prober.go:29] interesting pod/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 16:05:53.091078 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:53.091044 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 16:05:54.090018 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:54.089388 5476 patch_prober.go:29] interesting pod/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 02 16:05:54.090018 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:54.089485 5476 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 02 16:05:54.708520 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:54.708472 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:55.086085 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:55.086011 5476 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:05:55.086976 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:05:55.086960 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:06:20.145602 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:06:20.145564 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:06:20.292658 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:20.292627 5476 scope.go:122] "RemoveContainer" containerID="289b9c03f64ac6fdd278d1fd729a9c1cc4e497fc0a11b9d5abf5c62dfc363fad" Nov 02 16:06:22.882594 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:22.882558 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:06:23.453615 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:23.453580 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc"] Nov 02 16:06:23.453809 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:23.453788 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" containerID="cri-o://9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128" gracePeriod=30 Nov 02 16:06:25.088901 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:25.088870 5476 patch_prober.go:29] interesting pod/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 16:06:25.089334 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:25.088916 5476 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 16:06:28.706664 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.706631 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:06:28.820157 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820068 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-envoy\") pod \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " Nov 02 16:06:28.820157 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820121 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-token\" (UniqueName: \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-token\") pod \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " Nov 02 16:06:28.820157 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820155 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-credential-socket\") pod \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " Nov 02 16:06:28.820462 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820191 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-socket\") pod \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " Nov 02 16:06:28.820462 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820212 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-certs\") pod \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " Nov 02 16:06:28.820462 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820244 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istiod-ca-cert\") pod \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " Nov 02 16:06:28.820462 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820283 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-podinfo\") pod \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " Nov 02 16:06:28.820462 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820324 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-kube-api-access-v7gtx\" (UniqueName: \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-kube-api-access-v7gtx\") pod \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " Nov 02 16:06:28.820462 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820364 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-data\") pod \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\" (UID: \"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb\") " Nov 02 16:06:28.820792 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820518 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-socket" pod "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" (UID: "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:06:28.820792 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820525 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-credential-socket" pod "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" (UID: "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:06:28.820792 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820545 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-certs" pod "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" (UID: "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:06:28.820792 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820740 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-data" pod "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" (UID: "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:06:28.820960 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820815 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istiod-ca-cert" pod "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" (UID: "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:06:28.820960 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820866 5476 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-data\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:28.820960 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820882 5476 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-credential-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:28.820960 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820899 5476 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-socket\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:28.820960 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.820909 5476 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-workload-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:28.822383 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.822358 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-envoy" pod "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" (UID: "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:06:28.822636 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.822620 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-kube-api-access-v7gtx" pod "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" (UID: "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb"). InnerVolumeSpecName "kube-api-access-v7gtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:06:28.822949 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.822922 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-token" pod "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" (UID: "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:06:28.822949 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.822919 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-podinfo" pod "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" (UID: "d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 16:06:28.921301 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.921261 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7gtx\" (UniqueName: \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-kube-api-access-v7gtx\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:28.921301 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.921298 5476 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-envoy\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:28.921301 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.921308 5476 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-token\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:28.921545 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.921320 5476 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istiod-ca-cert\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:28.921545 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:28.921329 5476 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb-istio-podinfo\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:29.097712 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.097623 5476 generic.go:356] "Generic (PLEG): container finished" podID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerID="9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128" exitCode=0 Nov 02 16:06:29.097712 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.097664 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" event={"ID":"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb","Type":"ContainerDied","Data":"9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128"} Nov 02 16:06:29.097712 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.097673 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" Nov 02 16:06:29.097712 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.097710 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc" event={"ID":"d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb","Type":"ContainerDied","Data":"376bb6c645bdd773461c6acf0db96bd1afbf56db5118dd345036da23226983aa"} Nov 02 16:06:29.098118 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.097734 5476 scope.go:122] "RemoveContainer" containerID="9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128" Nov 02 16:06:29.107635 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.107615 5476 scope.go:122] "RemoveContainer" containerID="17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e" Nov 02 16:06:29.116052 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.116028 5476 scope.go:122] "RemoveContainer" containerID="9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128" Nov 02 16:06:29.116295 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:06:29.116276 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128\": container with ID starting with 9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128 not found: ID does not exist" containerID="9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128" Nov 02 16:06:29.116339 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.116305 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128"} err="failed to get container status \"9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128\": rpc error: code = NotFound desc = could not find container \"9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128\": container with ID starting with 9bd72bf0a5fdc95fab0e3f95e9c8c0c6cee05e6420087ac5ce5bff1e054eb128 not found: ID does not exist" Nov 02 16:06:29.116339 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.116324 5476 scope.go:122] "RemoveContainer" containerID="17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e" Nov 02 16:06:29.116579 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:06:29.116561 5476 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e\": container with ID starting with 17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e not found: ID does not exist" containerID="17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e" Nov 02 16:06:29.116628 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.116586 5476 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e"} err="failed to get container status \"17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e\": rpc error: code = NotFound desc = could not find container \"17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e\": container with ID starting with 17182586e6945f6b1aecf1175e24f1db698f330265aa088e3725f6390511af8e not found: ID does not exist" Nov 02 16:06:29.252983 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.252949 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc"] Nov 02 16:06:29.299540 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.299504 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f22ac84e-openshift-default-9df8fb875-8p5mc"] Nov 02 16:06:29.927492 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:29.927459 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" path="/var/lib/kubelet/pods/d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb/volumes" Nov 02 16:06:35.305838 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.305797 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z"] Nov 02 16:06:35.306456 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.306432 5476 cpu_manager.go:414] "containerMap: removing container" podUID="70fa749c-b6b9-4849-a297-5c3692ccd58d" containerName="maas-api" Nov 02 16:06:35.306561 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.306458 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70fa749c-b6b9-4849-a297-5c3692ccd58d" containerName="maas-api" Nov 02 16:06:35.306561 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.306473 5476 cpu_manager.go:414] "containerMap: removing container" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" Nov 02 16:06:35.306561 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.306481 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" Nov 02 16:06:35.306561 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.306493 5476 cpu_manager.go:414] "containerMap: removing container" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" Nov 02 16:06:35.306561 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.306501 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" Nov 02 16:06:35.306822 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.306626 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="70fa749c-b6b9-4849-a297-5c3692ccd58d" containerName="maas-api" Nov 02 16:06:35.306822 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.306639 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" Nov 02 16:06:35.306822 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.306650 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b9a109-b109-4d55-bbac-6f9b8b7bc1bb" containerName="istio-proxy" Nov 02 16:06:35.311670 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.311647 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.363438 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.363384 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-715ca8-serving-cert\"" Nov 02 16:06:35.366143 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.366120 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z"] Nov 02 16:06:35.384316 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.384286 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnrv7\" (UniqueName: \"kubernetes.io/projected/4a82c4d3-f83b-42dc-8e69-a1b993e068db-kube-api-access-dnrv7\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.384493 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.384331 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a82c4d3-f83b-42dc-8e69-a1b993e068db-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.384493 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.384410 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.384608 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.384582 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-metrics-certs\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.485917 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.485879 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnrv7\" (UniqueName: \"kubernetes.io/projected/4a82c4d3-f83b-42dc-8e69-a1b993e068db-kube-api-access-dnrv7\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.486144 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.485931 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a82c4d3-f83b-42dc-8e69-a1b993e068db-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.486144 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.485960 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.486144 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:06:35.486090 5476 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-serving-cert: secret "maas-api-e2e-authcfg-http-715ca8-serving-cert" not found Nov 02 16:06:35.486144 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.486112 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-metrics-certs\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.486144 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:06:35.486146 5476 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-maas-api-tls podName:4a82c4d3-f83b-42dc-8e69-a1b993e068db nodeName:}" failed. No retries permitted until 2026-11-02 16:06:35.986128457 +0000 UTC m=+4098.189006495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-maas-api-tls") pod "maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" (UID: "4a82c4d3-f83b-42dc-8e69-a1b993e068db") : secret "maas-api-e2e-authcfg-http-715ca8-serving-cert" not found Nov 02 16:06:35.486707 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.486683 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a82c4d3-f83b-42dc-8e69-a1b993e068db-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.488478 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.488456 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-metrics-certs\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.544152 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.544107 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnrv7\" (UniqueName: \"kubernetes.io/projected/4a82c4d3-f83b-42dc-8e69-a1b993e068db-kube-api-access-dnrv7\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.992984 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.992952 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:35.995426 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:35.995407 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:36.223899 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:36.223871 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:36.593964 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:36.593879 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z"] Nov 02 16:06:36.599807 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:06:36.599779 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a82c4d3_f83b_42dc_8e69_a1b993e068db.slice/crio-fdf077099d9fdb3a0f929d9b3f13ff079f174485677d64d1875ef870b4785fc7 WatchSource:0}: Error finding container fdf077099d9fdb3a0f929d9b3f13ff079f174485677d64d1875ef870b4785fc7: Status 404 returned error can't find the container with id fdf077099d9fdb3a0f929d9b3f13ff079f174485677d64d1875ef870b4785fc7 Nov 02 16:06:37.147752 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:37.147717 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" event={"ID":"4a82c4d3-f83b-42dc-8e69-a1b993e068db","Type":"ContainerStarted","Data":"fdf077099d9fdb3a0f929d9b3f13ff079f174485677d64d1875ef870b4785fc7"} Nov 02 16:06:38.154988 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:38.154949 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" event={"ID":"4a82c4d3-f83b-42dc-8e69-a1b993e068db","Type":"ContainerStarted","Data":"a1d71fdf3776ff5f0bd36d717fb194497b1c378247f448a4a71aabfadc8c6b51"} Nov 02 16:06:38.155421 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:38.155031 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:38.266631 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:38.266577 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" podStartSLOduration=2.436693492 podStartE2EDuration="3.266562386s" totalImagesPullingTime="829.868894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:06:38.265686579 +0000 UTC m=+4100.468564640" watchObservedRunningTime="2026-11-02 16:06:38.266562386 +0000 UTC m=+4100.469440446" Nov 02 16:06:43.088802 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:43.088765 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:06:44.166433 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:06:44.166400 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:06:50.146262 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:06:50.146220 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:07:20.145955 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:07:20.145915 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:07:20.321245 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:07:20.321214 5476 scope.go:122] "RemoveContainer" containerID="68b6858d2ca4b5fd09dfb2c2b7db0b473493f4ec7e31db465f87347e9e7fa0ce" Nov 02 16:07:50.146126 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:07:50.146040 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:08:18.908708 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:18.908676 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 16:08:19.005566 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:19.005537 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qpkcp_449bb762-da7a-4a88-9796-1a1ef23e7c8c/ovn-acl-logging/0.log" Nov 02 16:08:20.146678 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:08:20.146640 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:08:37.604138 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.604101 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z"] Nov 02 16:08:37.604558 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.604304 5476 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" podUID="4a82c4d3-f83b-42dc-8e69-a1b993e068db" containerName="maas-api" containerID="cri-o://a1d71fdf3776ff5f0bd36d717fb194497b1c378247f448a4a71aabfadc8c6b51" gracePeriod=30 Nov 02 16:08:37.811792 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.811637 5476 generic.go:356] "Generic (PLEG): container finished" podID="4a82c4d3-f83b-42dc-8e69-a1b993e068db" containerID="a1d71fdf3776ff5f0bd36d717fb194497b1c378247f448a4a71aabfadc8c6b51" exitCode=0 Nov 02 16:08:37.812185 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.812078 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" event={"ID":"4a82c4d3-f83b-42dc-8e69-a1b993e068db","Type":"ContainerDied","Data":"a1d71fdf3776ff5f0bd36d717fb194497b1c378247f448a4a71aabfadc8c6b51"} Nov 02 16:08:37.859382 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.859324 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:08:37.984319 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.984291 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-metrics-certs\") pod \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " Nov 02 16:08:37.984654 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.984633 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-maas-api-tls\") pod \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " Nov 02 16:08:37.984843 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.984827 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a82c4d3-f83b-42dc-8e69-a1b993e068db-kube-api-access-dnrv7\" (UniqueName: \"kubernetes.io/projected/4a82c4d3-f83b-42dc-8e69-a1b993e068db-kube-api-access-dnrv7\") pod \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " Nov 02 16:08:37.985465 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.985439 5476 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a82c4d3-f83b-42dc-8e69-a1b993e068db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a82c4d3-f83b-42dc-8e69-a1b993e068db-openshift-service-ca\") pod \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\" (UID: \"4a82c4d3-f83b-42dc-8e69-a1b993e068db\") " Nov 02 16:08:37.986221 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.986078 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a82c4d3-f83b-42dc-8e69-a1b993e068db-openshift-service-ca" pod "4a82c4d3-f83b-42dc-8e69-a1b993e068db" (UID: "4a82c4d3-f83b-42dc-8e69-a1b993e068db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:08:37.987162 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.987131 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-maas-api-tls" pod "4a82c4d3-f83b-42dc-8e69-a1b993e068db" (UID: "4a82c4d3-f83b-42dc-8e69-a1b993e068db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:08:37.987162 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.987149 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-metrics-certs" pod "4a82c4d3-f83b-42dc-8e69-a1b993e068db" (UID: "4a82c4d3-f83b-42dc-8e69-a1b993e068db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:08:37.988483 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:37.988226 5476 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a82c4d3-f83b-42dc-8e69-a1b993e068db-kube-api-access-dnrv7" pod "4a82c4d3-f83b-42dc-8e69-a1b993e068db" (UID: "4a82c4d3-f83b-42dc-8e69-a1b993e068db"). InnerVolumeSpecName "kube-api-access-dnrv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:08:38.086713 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:38.086677 5476 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-metrics-certs\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:38.086713 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:38.086704 5476 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a82c4d3-f83b-42dc-8e69-a1b993e068db-maas-api-tls\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:38.086713 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:38.086715 5476 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnrv7\" (UniqueName: \"kubernetes.io/projected/4a82c4d3-f83b-42dc-8e69-a1b993e068db-kube-api-access-dnrv7\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:38.087033 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:38.086724 5476 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a82c4d3-f83b-42dc-8e69-a1b993e068db-openshift-service-ca\") on node \"ip-10-0-142-141.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:38.818867 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:38.818834 5476 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" Nov 02 16:08:38.818867 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:38.818849 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z" event={"ID":"4a82c4d3-f83b-42dc-8e69-a1b993e068db","Type":"ContainerDied","Data":"fdf077099d9fdb3a0f929d9b3f13ff079f174485677d64d1875ef870b4785fc7"} Nov 02 16:08:38.819403 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:38.818946 5476 scope.go:122] "RemoveContainer" containerID="a1d71fdf3776ff5f0bd36d717fb194497b1c378247f448a4a71aabfadc8c6b51" Nov 02 16:08:38.973363 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:38.973329 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z"] Nov 02 16:08:39.019840 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:39.019808 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-715ca8-6cfc96c9df-dpb4z"] Nov 02 16:08:39.921371 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:39.921340 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a82c4d3-f83b-42dc-8e69-a1b993e068db" path="/var/lib/kubelet/pods/4a82c4d3-f83b-42dc-8e69-a1b993e068db/volumes" Nov 02 16:08:42.128690 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:42.128656 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:08:50.146077 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:08:50.146040 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:08:55.176401 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:08:55.176367 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:09:05.484919 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:09:05.484836 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:09:20.059697 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:09:20.059662 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f"] Nov 02 16:09:20.105462 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:09:20.105383 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qqk8f"] Nov 02 16:09:20.147493 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:09:20.147457 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:09:21.921301 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:09:21.921267 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0" path="/var/lib/kubelet/pods/ac6d3822-ee5e-4b97-bc87-0dc1c15e89d0/volumes" Nov 02 16:09:26.563257 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:09:26.563226 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:09:40.051703 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:09:40.051668 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc"] Nov 02 16:09:40.097548 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:09:40.097520 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-vhtjc"] Nov 02 16:09:41.920857 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:09:41.920822 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4780db8d-0d1d-41ff-8495-eef08187c8c9" path="/var/lib/kubelet/pods/4780db8d-0d1d-41ff-8495-eef08187c8c9/volumes" Nov 02 16:09:50.147754 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:09:50.147717 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:09:58.623903 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:09:58.623867 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56fb547cfb-zdfhv_5015c227-7a33-4970-86c9-a1a88274a726/authorino/0.log" Nov 02 16:09:58.870020 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:09:58.869975 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-7dljq"] Nov 02 16:10:06.045937 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:06.045899 5476 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq"] Nov 02 16:10:06.072073 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:06.072042 5476 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa999e8-ws2bq"] Nov 02 16:10:07.921615 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:07.921571 5476 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9457f436-daff-49ea-8158-da6b5f218b3c" path="/var/lib/kubelet/pods/9457f436-daff-49ea-8158-da6b5f218b3c/volumes" Nov 02 16:10:09.535464 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:09.535433 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pmdck_3726b436-7706-438d-90a2-d804726c1399/manager/0.log" Nov 02 16:10:09.818925 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:09.818857 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-h9cpc_a1b0feb9-f6d8-4593-a167-6f1b32daafe7/manager/0.log" Nov 02 16:10:11.265508 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:11.265478 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cn7hn_351c7a71-9451-4c65-9db0-a5d8d5d24c39/server/0.log" Nov 02 16:10:12.565444 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:12.565415 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57678c84d7-ggjmr_9c4e57a0-9bee-4b65-9b96-934bb2f16430/maas-api/0.log" Nov 02 16:10:15.339437 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:15.339405 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56fb547cfb-zdfhv_5015c227-7a33-4970-86c9-a1a88274a726/authorino/0.log" Nov 02 16:10:16.247424 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:16.247394 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-k4hl6_a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29/kuadrant-console-plugin/0.log" Nov 02 16:10:16.796789 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:16.796762 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7dljq_98f8fe94-c8fe-44ee-8a08-1db2c5489d5f/limitador/0.log" Nov 02 16:10:17.015938 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:17.015906 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cgq8m_73aedbe1-24c7-4641-bf3f-0f6b9c20688a/manager/0.log" Nov 02 16:10:18.381889 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:18.381863 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848b57c455-hljd7_741af07b-5653-4e7c-969c-469b7ada179b/kube-auth-proxy/0.log" Nov 02 16:10:18.838966 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:18.838893 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-256hx_a71c6c9c-6820-4ced-a069-a20e4dbe3841/istio-proxy/0.log" Nov 02 16:10:19.406263 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:19.406225 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-kk2m9_afd13c1e-9b91-4be5-92ed-ceea14f0203c/istio-proxy/0.log" Nov 02 16:10:20.146013 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:10:20.145962 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:10:20.374871 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:20.374838 5476 scope.go:122] "RemoveContainer" containerID="96e04ab9196c498367ba98267719a03ede9353b766984cfd6e44c82dc1f38926" Nov 02 16:10:20.997028 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:20.996985 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-wmnbp_9ee63d99-1900-45ed-a48d-64d2734fef42/istio-proxy/0.log" Nov 02 16:10:21.450754 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:21.450720 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s_be550f37-1beb-48d2-90cd-a0b7d8860fd4/istio-proxy/0.log" Nov 02 16:10:23.967659 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:23.967619 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx_78786688-ccde-4456-8b9e-9cd8ce62ceba/main/0.log" Nov 02 16:10:24.204610 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:24.204572 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-rvfxv_b2f3ed9a-28e9-48ad-8ecb-a03c3b779928/main/0.log" Nov 02 16:10:24.683359 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:24.683329 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj_4350e014-b37d-459a-a887-79d72ed3ed3c/main/0.log" Nov 02 16:10:25.568673 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:25.568639 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56fb547cfb-zdfhv_5015c227-7a33-4970-86c9-a1a88274a726/authorino/0.log" Nov 02 16:10:35.039096 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:35.039018 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-pmdck_3726b436-7706-438d-90a2-d804726c1399/manager/0.log" Nov 02 16:10:35.314045 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:35.313959 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-h9cpc_a1b0feb9-f6d8-4593-a167-6f1b32daafe7/manager/0.log" Nov 02 16:10:36.294652 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:36.294623 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cn7hn_351c7a71-9451-4c65-9db0-a5d8d5d24c39/server/0.log" Nov 02 16:10:37.362119 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:37.362075 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-57678c84d7-ggjmr_9c4e57a0-9bee-4b65-9b96-934bb2f16430/maas-api/0.log" Nov 02 16:10:39.757648 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:39.757617 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56fb547cfb-zdfhv_5015c227-7a33-4970-86c9-a1a88274a726/authorino/0.log" Nov 02 16:10:40.649982 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:40.649951 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-k4hl6_a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29/kuadrant-console-plugin/0.log" Nov 02 16:10:41.122585 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:41.122557 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7dljq_98f8fe94-c8fe-44ee-8a08-1db2c5489d5f/limitador/0.log" Nov 02 16:10:41.353489 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:41.353452 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cgq8m_73aedbe1-24c7-4641-bf3f-0f6b9c20688a/manager/0.log" Nov 02 16:10:42.518007 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:42.517976 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848b57c455-hljd7_741af07b-5653-4e7c-969c-469b7ada179b/kube-auth-proxy/0.log" Nov 02 16:10:42.962548 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:42.962516 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-256hx_a71c6c9c-6820-4ced-a069-a20e4dbe3841/istio-proxy/0.log" Nov 02 16:10:43.466444 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:43.466413 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-kk2m9_afd13c1e-9b91-4be5-92ed-ceea14f0203c/istio-proxy/0.log" Nov 02 16:10:44.944490 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:44.944462 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-wmnbp_9ee63d99-1900-45ed-a48d-64d2734fef42/istio-proxy/0.log" Nov 02 16:10:45.384289 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:45.384247 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-86b2s_be550f37-1beb-48d2-90cd-a0b7d8860fd4/istio-proxy/0.log" Nov 02 16:10:47.627531 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:47.627502 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rmcfx_78786688-ccde-4456-8b9e-9cd8ce62ceba/main/0.log" Nov 02 16:10:47.895713 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:47.895639 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-rvfxv_b2f3ed9a-28e9-48ad-8ecb-a03c3b779928/main/0.log" Nov 02 16:10:48.373437 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:10:48.373406 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-qtnpj_4350e014-b37d-459a-a887-79d72ed3ed3c/main/0.log" Nov 02 16:10:50.145971 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:10:50.145935 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:11:20.145881 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:11:20.145842 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:11:20.397221 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:11:20.397143 5476 scope.go:122] "RemoveContainer" containerID="d32748506a8442ba2796c26d93019163ff93546f45b31e3dfd13888112d70be6" Nov 02 16:11:20.406188 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:11:20.406165 5476 scope.go:122] "RemoveContainer" containerID="6f63fc702777431796bc3b1fa75274c38267eb0405ae91e5ec4a98ef30d6dd3d" Nov 02 16:11:50.146310 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:11:50.146263 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:12:04.519445 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:04.519372 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vjfjv_4bfe09e4-8fce-4149-a9fb-0a222124f121/global-pull-secret-syncer/0.log" Nov 02 16:12:07.337838 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:07.337796 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xp4n5_efc5a077-d018-4792-8551-b9c3594967dd/konnectivity-agent/0.log" Nov 02 16:12:08.505369 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:08.505333 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-141.us-west-2.compute.internal_5d1881377e4cd46a1e5cc5db6f951a84/haproxy/0.log" Nov 02 16:12:13.375826 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:13.375795 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56fb547cfb-zdfhv_5015c227-7a33-4970-86c9-a1a88274a726/authorino/0.log" Nov 02 16:12:14.533057 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:14.533025 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-k4hl6_a7a05c4e-b333-4e50-b2ad-0cdfa09bdf29/kuadrant-console-plugin/0.log" Nov 02 16:12:15.000949 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.000919 5476 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh"] Nov 02 16:12:15.001336 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.001324 5476 cpu_manager.go:414] "containerMap: removing container" podUID="4a82c4d3-f83b-42dc-8e69-a1b993e068db" containerName="maas-api" Nov 02 16:12:15.001384 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.001338 5476 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a82c4d3-f83b-42dc-8e69-a1b993e068db" containerName="maas-api" Nov 02 16:12:15.001421 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.001417 5476 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a82c4d3-f83b-42dc-8e69-a1b993e068db" containerName="maas-api" Nov 02 16:12:15.004593 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.004574 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.072202 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.072178 5476 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f2cdf\"/\"default-dockercfg-bcp6z\"" Nov 02 16:12:15.072202 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.072182 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2cdf\"/\"openshift-service-ca.crt\"" Nov 02 16:12:15.072202 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.072182 5476 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2cdf\"/\"kube-root-ca.crt\"" Nov 02 16:12:15.095746 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.095721 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-lib-modules\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.095866 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.095755 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csl6c\" (UniqueName: \"kubernetes.io/projected/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-kube-api-access-csl6c\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.095910 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.095889 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-sys\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.095948 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.095927 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-podres\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.096075 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.096057 5476 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-proc\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.101987 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.101962 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh"] Nov 02 16:12:15.197392 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.197364 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-proc\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.197532 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.197412 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-lib-modules\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.197532 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.197432 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csl6c\" (UniqueName: \"kubernetes.io/projected/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-kube-api-access-csl6c\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.197532 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.197467 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-sys\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.197532 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.197473 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-proc\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.197532 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.197495 5476 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-podres\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.197718 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.197554 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-lib-modules\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.197718 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.197556 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-sys\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.197718 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.197625 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-podres\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.254743 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.254686 5476 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csl6c\" (UniqueName: \"kubernetes.io/projected/84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53-kube-api-access-csl6c\") pod \"perf-node-gather-daemonset-jwvlh\" (UID: \"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.317374 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.317350 5476 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:15.326698 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.326677 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-7dljq_98f8fe94-c8fe-44ee-8a08-1db2c5489d5f/limitador/0.log" Nov 02 16:12:15.686417 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.686387 5476 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh"] Nov 02 16:12:15.692524 ip-10-0-142-141 kubenswrapper[5476]: W1102 16:12:15.692498 5476 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod84ed91c7_ab49_4b8f_a5fd_bdf99aeabf53.slice/crio-1ea2c420d427308b5db5078a4da6d72f9609e9f6b3ef599d964f9abb5e3aa3cc WatchSource:0}: Error finding container 1ea2c420d427308b5db5078a4da6d72f9609e9f6b3ef599d964f9abb5e3aa3cc: Status 404 returned error can't find the container with id 1ea2c420d427308b5db5078a4da6d72f9609e9f6b3ef599d964f9abb5e3aa3cc Nov 02 16:12:15.761685 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:15.761662 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-cgq8m_73aedbe1-24c7-4641-bf3f-0f6b9c20688a/manager/0.log" Nov 02 16:12:16.166064 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:16.166030 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" event={"ID":"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53","Type":"ContainerStarted","Data":"f14c05672a96e42080c2b264601748978201f1f49110a96de7a4a190814b54d0"} Nov 02 16:12:16.166234 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:16.166077 5476 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" event={"ID":"84ed91c7-ab49-4b8f-a5fd-bdf99aeabf53","Type":"ContainerStarted","Data":"1ea2c420d427308b5db5078a4da6d72f9609e9f6b3ef599d964f9abb5e3aa3cc"} Nov 02 16:12:16.166234 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:16.166159 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:16.318648 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:16.318599 5476 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" podStartSLOduration=2.318585983 podStartE2EDuration="2.318585983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 16:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:12:16.276575205 +0000 UTC m=+4438.479453265" watchObservedRunningTime="2026-11-02 16:12:16.318585983 +0000 UTC m=+4438.521464043" Nov 02 16:12:18.348283 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:18.348249 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc7b9133-0f4a-4e46-bfe6-7282b3d26e90/alertmanager/0.log" Nov 02 16:12:18.751440 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:18.751409 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc7b9133-0f4a-4e46-bfe6-7282b3d26e90/config-reloader/0.log" Nov 02 16:12:19.161094 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:19.161051 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc7b9133-0f4a-4e46-bfe6-7282b3d26e90/kube-rbac-proxy-web/0.log" Nov 02 16:12:19.557234 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:19.557149 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc7b9133-0f4a-4e46-bfe6-7282b3d26e90/kube-rbac-proxy/0.log" Nov 02 16:12:19.952568 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:19.952541 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc7b9133-0f4a-4e46-bfe6-7282b3d26e90/kube-rbac-proxy-metric/0.log" Nov 02 16:12:20.145471 ip-10-0-142-141 kubenswrapper[5476]: E1102 16:12:20.145434 5476 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lgnjt" containerName="extractor" Nov 02 16:12:20.347178 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:20.347093 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc7b9133-0f4a-4e46-bfe6-7282b3d26e90/prom-label-proxy/0.log" Nov 02 16:12:20.742903 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:20.742859 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cc7b9133-0f4a-4e46-bfe6-7282b3d26e90/init-config-reloader/0.log" Nov 02 16:12:22.182360 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:22.182328 5476 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-jwvlh" Nov 02 16:12:24.687049 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:24.687017 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qzm49_d186e7b3-d2fa-4f5b-bd31-ee633608c8db/node-exporter/0.log" Nov 02 16:12:25.080356 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:25.080266 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qzm49_d186e7b3-d2fa-4f5b-bd31-ee633608c8db/kube-rbac-proxy/0.log" Nov 02 16:12:25.474569 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:25.474540 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qzm49_d186e7b3-d2fa-4f5b-bd31-ee633608c8db/init-textfile/0.log" Nov 02 16:12:28.283261 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:28.283231 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_922b435d-9e36-4f79-b74d-c177443df3f0/prometheus/0.log" Nov 02 16:12:28.675812 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:28.675782 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_922b435d-9e36-4f79-b74d-c177443df3f0/config-reloader/0.log" Nov 02 16:12:29.075667 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:29.075570 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_922b435d-9e36-4f79-b74d-c177443df3f0/thanos-sidecar/0.log" Nov 02 16:12:29.476319 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:29.476296 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_922b435d-9e36-4f79-b74d-c177443df3f0/kube-rbac-proxy-web/0.log" Nov 02 16:12:29.871144 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:29.871120 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_922b435d-9e36-4f79-b74d-c177443df3f0/kube-rbac-proxy/0.log" Nov 02 16:12:30.268274 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:30.268192 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_922b435d-9e36-4f79-b74d-c177443df3f0/kube-rbac-proxy-thanos/0.log" Nov 02 16:12:30.662938 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:30.662909 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_922b435d-9e36-4f79-b74d-c177443df3f0/init-config-reloader/0.log" Nov 02 16:12:31.068713 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:31.068621 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-kwcld_d1af9240-8442-4911-b1d9-5570aea28abf/prometheus-operator/0.log" Nov 02 16:12:31.465492 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:31.465463 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-kwcld_d1af9240-8442-4911-b1d9-5570aea28abf/kube-rbac-proxy/0.log" Nov 02 16:12:32.292749 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:32.292718 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7899fb9795-f778g_5e8592fc-f083-4eb8-b934-517c47f123c4/telemeter-client/0.log" Nov 02 16:12:32.692556 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:32.692528 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7899fb9795-f778g_5e8592fc-f083-4eb8-b934-517c47f123c4/reload/0.log" Nov 02 16:12:33.113405 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:33.113381 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7899fb9795-f778g_5e8592fc-f083-4eb8-b934-517c47f123c4/kube-rbac-proxy/0.log" Nov 02 16:12:38.291138 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:38.291111 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848b57c455-hljd7_741af07b-5653-4e7c-969c-469b7ada179b/kube-auth-proxy/0.log" Nov 02 16:12:38.697140 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:38.697110 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-256hx_a71c6c9c-6820-4ced-a069-a20e4dbe3841/istio-proxy/0.log" Nov 02 16:12:39.103018 ip-10-0-142-141 kubenswrapper[5476]: I1102 16:12:39.102925 5476 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-kk2m9_afd13c1e-9b91-4be5-92ed-ceea14f0203c/istio-proxy/0.log"